I think handhelds have always been merely a compromise of convenience. You deal with a less featured console for the purposes of portability. Nobody wanted black & white scaled down NES games on the Game Boy. They tolerated it merely because that was the limitations of the time and having gaming on the go was worth it. The "on the go" nature also made handheld games more disposable than consoles. A good proof of that are in Western handhelds games. How many truly great handheld games are not Japanese? Companies like EA and Activision just treat handhelds games as some half-assed product to provide a mindless distraction for the bus rider during his commute.
I think for that disposable time-waster market, the smartphones have won. Yeah they're not that great for games but that doesn't matter. This isn't about games in the way dedicated hardcore games see them but in the way a casual mainstream user sees them. Complaining that the iPhone is not for real games is like complaining that the crossword is not literature - it isn't supposed to be.
But what about the market of dedicated gamers that played console-quality games like Link's Awakening on their Game Boy and are looking forward to Fire Emblem on their 3DS? That is still a valid market but is not the huge market that ate up the DS. A big reason why handhelds and consoles were different systems with incompatible games was because of technological limitations. Nintendo couldn't make a handheld that could play NES cartridges and make it affordable and have decent battery life (and those big NES cartridges didn't help) so they came up with a seperate format. But without any technological restriction why would these be different formats? This is why the PSP and Vita are jokes. So you get console-comparible games on a handheld. Sounds good, except that why isn't the PS Vita just a portable PS3? Why are the games different? It isn't like this with other media. I can watch the same movies on a portable DVD player that I can at home. I can listen to the same music on my iPod as I do on my stereo at home. That's what people want and it's what makes sense. The only reasons to not have this is because it isn't technologically feasible or because of outright corporate greed.
So the future is that there is no difference between handheld and console games. They're just videogames and you can choose to have a handheld or console model that plays them, like how in the past you chose to buy a Discman or a standalone CD player and though both played the same albums. Console generations are getting longer as the obvious benefits of hardware improvements are less noticable to the consumer and thus less marketable. In that time such hardware could become cheap enough and energy efficient enough to put in a portable model, no differently than today's computer hardware being in tomorrow's laptops. The move to digital distribution also removes the need to have pocket sized media.
The future of handhelds is that you have your handheld and your console and they can both access the same cloud were you keep your games and saves and you play games in whatever manner best suits your lifestyle. There is no differentiation. We just have "videogame systems".
Though I also think we might lose the dedicated videogame market outright, not because there are not enough gamers to buy such games. No, the problem will be that the mainstream casual market will be so lucrative that less companies will bother to make core games. They'll want the big mainstream dollars and will go for that and only a small amount of dedicated companies will go after the enthusiasts (think like hobby companies that make model trains and such), and that will have the avalanche effect of making core gaming more and more niche until it is not profitable for anyone. Hopefully it does not get to that point but if it did, then we wouldn't see any dedicated videogame systems.