Yeah, this is also a tough choice for me. I've got a little belt-clip holder for my 3DS, so I carry it around with me everywhere and can pull it out very easily. With the 3DS and the XL, it's basically just the difference between pulling it off my hip holster and pulling it out of my bag. Tiny difference, but it really affects which one I'll take out. Which makes me want to keep my 3DS, but then, I like that bigger screen. A lot.
But the XL is really not pocketable, not on a regular basis, so unless you're going to wear cargo pants every single day, I would not bank on that. But with the current 3DS, remember that its footprint is the same size (and in some cases, smaller) than most Android smartphones, especially some of those HTC ones.