Depends what you like... here's the ones that I ended up playing a lot:
Dragon Quest 8: My absolute favorite game of this generation. A long, beautiful traditional RPG with memorable characters, a great animation style, and a lot of fun. Don't just do the main quests... explore around a little. Be sure to visit Le Puff Puff Club. That's easily the loudest I've ever laughed at a video game (nobody who's played it give it away, it's way funnier to go into it totally blind), even moreso than some of the stuff in Mario&Luigi Superstar Saga.
.hack// 1-4: Action RPG with some great tie-ins to the animes... make sure you can get a copy of 2 before you get too deep into 1, that's the toughest to find.
Xenosaga: Only buy these if you like long cutscenes. A lot of people get restless during a half hour cutscene and probably wouldn't enjoy the game. I love that sort of thing, personally.
GTA: San Andreas: For a game that's poked fun at so much, it really is impressively deep and well done. This is a game that gives you probably 80 hours of play time if you intend to do all the side-missions.
Dynasty Warriors 4: A lot of people find this game repetitive, personally I love it. If you're into it, this is the sort of game you'll be playing for months trying to unlock everything, then hunting around the Koei Japan website to put in your high scores in the challenge mode. I bought it over a year ago, it's been one of my most heavily played PS2 games in that time, and I'm still nowhere near unlocking everything. Great two-player co-op mode. There's nothing like it on the Gamecube...
Kingdom Hearts: A very well done game with a great story.
Star Ocean: Till the End of Time: A great story with a mediocre game attached. The storyline itself is fairly riveting, but I could never get past the ugly fight system to get very deep into it.
VF4: The sort of graphics the PS2 was supposed to deliver from the get go, VF4 is a great fighter that deserves some attention.