I think the main reasons why the Japanese Game Industry has declined to near-irrelevance are as follows:
Japanese pop culture is severely centered around milking one particular trend that manages to become widely popular, and their entertainment industry is extremely slow to adapt to change. As the rest of the world began to adopt new gaming conventions and styles of play near the end of the last console generation, the Japanese have gone gradually insular focusing on the core otaku still buying games with the styles and mechanics of the games they made in the PS1 and PS2 eras. I think a lot of Japanese companies are still reeling right now from the realization that what made them successful in previous console generations is no longer relevant (even increasingly within Japan as the otaku market continues to shrink), and they don't know how to adapt to new paradigms. They also have a tendency these days to make games that appeal to specific Japanese fetishes that aren't socially acceptable outside Japan, such as their current obsession with the dying moe fad.
Meanwhile, Western companies have continually innovated and incorporated new ideas and new ways at looking at games within their products. Unfortunately, Japanese companies don't seem to understand the reasons why certain features that have become prominent in Western games achieved their popularity. Seeing a trend they could potentially capitalize on, we've seen Japanese companies try to make "western-friendly games", and they've largely failed due to that lack of understanding.
Take for instance Dragon's Dogma, a recent JRPG from Capcom. In that game's case, Capcom looked at Western Role-Playing Games, and apparently determined that the key to their success was large open worlds with a heavy emphasis on travel and exploration. The problem is that they didn't understand that the key to the Open World design isn't just that the world is big and that there's places to go, but that exploring the world and going to these places is meaningful. You don't climb that mountain just for the sake of climbing that mountain. You climb it because you believe there could be something worth finding atop that mountain. And while journeying through these open worlds can be enjoyable, our time is valuable so we need some sort of easy fast travel system to cut down on tedium. Capcom didn't understand this, so Dragon's Dogma has a huge, largely empty world with an extremely restrictive fast travel system. They got the surface level concept right, but didn't understand the deeper philosophies that govern it so they botched the execution.
And there are any number of games where I've seen similar problems. I also saw something similar with Gravity Rush on the Vita, which also has a large open world to explore not unlike a Western game like Crackdown, but the world is largely empty with extremely basic exploration. Final Fantasy XIII-2 on the PS3 seems to want to steal the dialogue wheel from any given Bioware RPG, but the player has no meaningful agency within these dialogue wheels and they're too basic so it's pretty meaningless. For Japanese companies to find relevance in modern gaming, they need to stop designing around what was popular 10-15 years ago and adapt to how game design has evolved. That means understanding trends beyond surface level analysis.
And that doesn't mean that the Japanese have to essentially make Western games. But they do need to understand Western gaming so they can then make the necessary adjustments to make their games relevant while also maintaining their own unique flavor. Xenoblade shows how this possible, as does the Last Story from all accounts. Those games have adopted very Western design sensibilities, and yet I think Xenoblade at least has a very distinctive Japanese feel to it in terms of the story and atmosphere. I think Platinum's Vanquish is a good example of this as well, as it has a very Japanese kinetic feeling to it totally unique from your standard Western 3rd person shooter. Sega combined Japanese storytelling with turn-based strategy and 3rd person shooting to create something totally unique and revolutionary in Valkyria Chronicles (and then they turned around and clung to Japanese High School cliches for the first PSP sequel, which was nowhere near as well-received outside Japan).
Also, in the case of Japanese Role-Playing Games specifically, so many companies seem stuck in PlayStation 1 era scenario and character design sensibilities that have become tiring cliches at this point, cliches that desperately need to be retired by now.
I've also read that (due to how their development teams are structured) apparently the Japanese game production process is extremely inefficient compared to typical Western developers, leading to fewer quality products over a longer period of time. If I remember correctly it has something to do with hierarchy in management.