Why do certain games in Japan get dubbed "too difficult" for the North American market? I can understand reasons like Japanese specific anime/manga games, cultural differences, lack of funds and the game didin't do well. But I enjoy challenges. What are they basing the difficulty on and whom are they basing the difficulty against? Ninja Gaiden and Ninja Gaiden 2 were chaotic, but I think a lot of people enjoyed that. I found this article to be an interesting read. On a side note there are games like Soma Bringer for the DS where I can see no good reason at all for not releasing the game here in North America. The game was so good that a local, dedicated team of people have been translating it and have near completed it. However, the only way to play this game version is through a card such as the R4.
Great questions, although I don't have all the answers, I can give you some.
As you know, certain games often get released in Japan way ahead of the NA and EU markets, which is why sometimes Jpse publishers use this as a "testing phase" for their games. If they get feedback from fans that they loved the game BUT it was way too difficult in certain parts, they can then make slight tweaks to the game before they release the English version. Some people may see this as "dumbing it down", but in fact they are possibly trying to make it less annoying for the western market, who may not have the patience that Jpse gamers do (see my latest Editorial for more on that) and it may get poorer reviews as a result.
One famous example is Majora's Mask where the original version didn't have the "save owls", which made the game a lot more difficult and time-consuming, compared to the tweaked western version which did.
In fact I'd argue the opposite, that Jpse gamers prefer games slightly easier than us. Take Yoshi's Story (N64) as an example, that was actually made more difficult because Nintendo thought it wasn't challenging enough, they even added extra collectables which affected the ending, something that us westerners like in our games. (apparently)
As you mentioned, the original Ninja Gaiden games were also a lot more difficult than their Japanese originals, doing crazy-harsh things like taking out continues and unfair damage from enemies.
As for games like Soma Bringer not coming to the west, your guess is as good as mine. As you probably know, it's usually down to financial reasons, not enough sales potential, don't want to waste money to translate the game and publish it abroad if it won't sell etc etc
Heck, even Nintendo did it with Sin and Punishment (which was in English anyway) and more recently the Starfy games, absolutely no reason for not bringing them over.
Talking of fan translations, I'm sure you're aware of famous examples like Mother 3 and the Fatal Frame 4 English patch, it's scary when fans go to more effort than the publishing houses themselves...
I wonder if there are any Japanese gamers that are frustrated that a certain western game doesn't get released in Japan!
The only example that comes to mind was Grand Theft Auto, but now they're seeing releases here thanks to Capcom.