Dropping physical media would be a retail suicide, as far as I'm concerned. Something Microsoft and Sony actually do well is dominate shelf space in stores like GameStop while Wii U takes up a sad corner. It would only be another nail in the coffin.
Think of all that Wii U and 3DS shelf space that could be amiibo and other Nintendo merchandise! Also, new game discs are a low margin anyway, so Wal-Mart and Target probably wouldn't mind carrying less of those. And GameStop sees the writing on the wall for used game discs.
For Nintendo in particular, they have always kept the size of their games very small relative to their peers. Only games like Bayonetta 2, Super Smash Bros. for Wii U, and Xenoblade Chronicles X break the 10/20 GB barrier.
This has me revisiting a daft idea I had that is a compromise between the all-digital future and retail stores. The idea is kiosks in the store that locally stored the data for a lot of video games. You can purchase games from the kiosk, have it loaded on to your special cartridge, USB stick, or (maybe special?) SD card faster than it would take to download it, and then bring that data back home to your system. Of course, there will be redundancies to ensure the purchase is tied to your Nintendo account; the locally stored data is for those with poor (or maybe no) internet connections. These kiosks can have chain-specific sales too.
I say this is daft because there are security issues, you still have a retailer as a middle-man, you will need to train retail staff to operate and maintain the kiosk, and this process is too complex compared to buying a disc or from the eShop.