The real reason is probably something closer to not wanting their controller to be completely lifted from the SNES controller conventions, and having symbols instead also is an easy way to have something unique to their brand of video game products.
That's the only thing I could think of but why not just go with 1, 2, 3, 4? Or other letters even?
The O/X = Yes/No thing I would of never thought of because I don't associate O/X that way. Thinking back to certain games and possibly some game shows, it makes sense now but I never would've thought of that as a kid because it just wasn't common enough. Square is supposed to represent paper thus meaning menus..what? That's just fucking over-thinking it, c'mon.. that can't be real.
Broodwars, thank you; from now on I'll always remember where Triangle is on the dualshock now that you've pointed out it points up. Oh wait, the picture i'm looking at is a bit angled so my perspective makes it seem like it's not pointing up, it's pointing at square instead, or is it pointing at circle?
Granted, if you grew up with either the NES or SNES, ABXY are easy to remember, but circle, square, x, triangle are just.. odd. To this day I can't memorize that thing. At least on Xbox I can remember to flip 'em, so even if the game or another person tell me to "press A!" I know to hit B. The second you give me a dualshock though, 20+ years of controller use goes out the window; no matter how many times I use the damn thing.
The only other controller to trip me up so bad was the Saturn controller, but there the problem either stemmed from my fingers not being used to the extra buttons, thus fumbling if I had to jump from A to Z - or the fact that my thumbs rested easier on BC/YZ and had to stretch out for X and A buttons (I still had kid hands back then).