I don't think it can be so simplified as to say that if Nintendo had used CDs for the Nintendo 64, it would have been the market leader for that generation. There are a lot of reasons why both companies and consumers chose the PlayStation so simply using CDs would not have been a magical cure-all. First of all, Nintendo charged higher royalties than both Sony and Sega, which would have still made the PlayStation a better choice than a CD Nintendo 64. Also, a lot of companies hated Nintendo for their controlling ways, and even though they were more lenient by the time the Nintendo 64 launched, the PlayStation had no restrictions. One such company was Square, who I believe publicly insulted Nintendo, and likely would have still released all their games exclusively on the PlayStation. Without Final Fantasy VII and all of Square's other titles, there's no way the CD Nintendo 64 would be the market leader, especially in Japan, though it probably would have ended up closer than it was. Then there's the controller, which, after Sony slapped on the two analogue sticks and rumble, most people considered it superior to Nintendo's three-pronged monstrosity (I'm not one of them though).
It's easy to look back and see why Nintendo should have chose CDs, but take a step back to 1995 and look at it from that perspective. Up to that point, no CD system had been very successful; the closest is the Sega CD which could maybe be considered a moderate success, but things like the Atari Jaguar, CD-i, 3DO, etc all failed miserably. Neither the Saturn or PlayStation were selling very well either, so it's easy to see why Nintendo would stick with the medium that had brought them much success up to that point.
What if the N64 was a cart/CD combo?
It almost was a combo system... sort of. The 64DD was revealed before the Nintendo 64 launched, and was originally scheduled to launch close to the N64's launch. Although the 64DD's disks were still far smaller than CDs, they could store more data than the N64's cartridges (at that time) and were cheaper to produce. Supposedly, part of the reason the add-on as created was to address concerns of low space and production costs, but I don't know if that's true or not. What I do know is, the 64DD had amazing potential, and if its hardware were integrated into the N64 itself and its disks were used instead of the cartridges, it might have fared a little better in the marketplace.