You don't need a GBA if you're only playing single-player. Whenever you enter something that's supposed to be on your GBA, a "virtual GBA" pops up on your screen, and you just play inside that with your regular controller. They couldn't do that in multiplayer, because it'd block the screen and get in everyone else's way. But in single-player it's just fine.
The port of LttP was only available on the GBA. And actually, the Four Swords on the GBA is different from the one on the Cube. The Cube's Four Swords is the sequel to the GBA's Four Swords. But the Cube's Four Swords game is much more enjoyable as a single-player game than the original was. I think you'll really enjoy it, even if it's just as a single-player game, and you haven't played the original.