Yeah. You *can* find them online for a little cheaper, with the standard size 64mb one going for as low as $40. But that's online -- Sony's MSRP is still $65-70, and you can check that against any retail store in your area, as they're stocked in the Camcorder and Digital Camera section.
Sony has been really headstrong about trying to get its Memory Stick into wider usage, because they know they can make a lot of money off the thing. But as they're the only ones who incorporate their technology, it's only really useful if you have a lot of Sony stuff.
You can find smaller memory sticks, but they're usually overpriced for what you get. They're fond of chucking 8mb ones in with their camcorder products, so they may very well include a memory stick with their PSP. But yeah, I can't imagine people filling up the freebie 8mb one and being stuck buying a 64mb card at Best Buy for $60.
I agree somewhat with what ActorJ says about how Sony isn't necessarily making a product to compete directly with the GB, but rather a product that's similar yet different enough to justify the new product. I could see it being popular if it's touted as a "portable console" instead of a hand-held, but, well, are people really going to buy an outdated console that isn't truly portable? The GBA:SP is selling MUCH better than the original GB, and I'm sure a lot of that is because of the sleek design and, perhaps most importantly, the incredibly small form factor. You can't fit a 4.5" screen in your pocket, let alone a 4.5" screen with a disk-slot and controller buttons attached.
I agree that the Playstation name is a good one for gaming right now, but that doesn't mean it'll translate into every facet of gaming. People may see the PSP, especially if it launches with PS1 games, as a gimmick; an attempt to cash in on old tech and get some of the GB money, with a larger screen and 3D. They'll say things like "I already own these games and play them on my PS2, why buy them again so I can play them in the kitchen on a smaller screen?" With a larger screen and overall size, people won't say "Wow, I can just put this in my pocket and I can play these games I liked wherever!" Hell, some people thought the original GBA was too big!
Cell phone companies are starting to emphasize the downloading of games to cellphones -- even Verizon has a new commercial with their "can you hear me now" guy playing crappy basketball. Why cell phones? Because they're convenient *and* they're small. People just slip cell phones in their pocket without really planning for it, and cell phones only really exploded when they got below a certain form factor. People *like* small things when they're taking them places. The GBA:SP fits in a pocket even with a few games, cos the cartridges are so small. Does anyone honestly think that the GBA:SP would sell better if the screen was twice as big? No one would take them anywhere unless they were already carrying a bag. I mean, look at Apple's iPod -- ultra-small!