I don't think the GameBoy was really "niche" before Pokemon. There were a lot of not just good, but big games on the system. Metroid 2. A couple of Mario games. Several Megaman and Castlevania titles. Four Square RPGs. I think the original GameBoy probably had more 3rd party support than the Sega Genesis. They even released the Super GameBoy on the SNES. All before Pokemon.
Pokemon was just "the next Tetris", and made for a powerful lead-in to the GameBoy Color.
IMO, the old fat GameBoy represented the pre-Pokemon days, and the GameBoy Pocket was for the later ones.
I do remember that almost nobody ever had any classic GameBoy systems or games to rent anywhere, but all the videogame selling places seemed to give the GameBoy as much shelf space as the NES and the Genesis (which was much more than something niche, like the TG-16).
I think the GameBoy was just "a little different" from regular videogames, but not "niche", because it was still big (not "reports on the news of teachers having to confiscate GameBoy units" big, but still big).
I remember the Game Gear being rather "niche" though. You had to go to a "serious" videogame place to even be able to buy stuff. It seemed to me that only the "hardcore" people even knew it existed (and always seemed mad at the popularity of the GameBoy, for "hindering progress", which never made sense to me).
Oh and, you usually can't really use the Saturn and Dreamcast as comparisons for anything. The Saturn supposedly actually did pretty well in Japan, but it got canned in America by Bernie Stolar so that they could concentrate on the launch of the Dreamcast. And then the Dreamcast just got unplugged in it's early life with severe abruptness.