There is no need to buy 3 unless you like the old style fast menus more, 4 has everything 3 got and more, a lot more cars and a lot more tracks, car handling is imo a lot better too in 4 especially if you are using a wheel.

Though I haven't played it Killer 7 should be a unique experience and Rez is nice too if you like trippy stuff.
Get Silent Hill 2 and God Of War also as popular as they are they are also very good.

The /v/ wiki has a good selection of great gems for pretty much any gaming plataform ever released (well, at least the relevant ones):

http://vsrecommendedgames.wikia.com/wiki/PlayStation_2

Once you go through all that, try out racketboy´s list:

http://www.racketboy.com/retro/sony/ps2/the-best-undiscovered-playstation-2-ps2-games

Manhunt and Manhunt 2!

Get stuff you can't get on superior platforms. Devil May Cry 1, God Hand, the .Hack series (I'm a fan xP), Final Fantasy 12, Timesplitters 1-2-3...you can get the HD remakes of other ones so it's not too important to grab them. :)

Other exclusives have been Level 5 games like Dark Cloud 1 and 2 and Rogue Galaxy. Legaia 2 Duel Saga was an enjoyable rpg for me, as was Radiata Stories. OH, definitely look into Mister Mosquito. Loved that game and there's nothing quite like it. ^^

King's Field is an excellent series. I remember being surprised at the characters in AC, as that was the first time the characters had actual faces. KF was a bit brutal at times, but it was definitely one of the most interesting FP RPGs of its time and it still holds up now. I would say KF fits into a similar niche as the Japanese Wizardry games (not including the most recent which, although having Manga-style character graphics is very close in spirit to Proving Grounds of the Mad Overlord.)
In any case, KF is a classic that was very big in Japan but overlooked in the West.

If you are a fan of the TV show "24", you might like "24: The Game" (PS2 exclusive.) If you are not a fan of the show, you will most likely hate it.

I'm a hardcore fan, and I loved it. It was like playing a new season of the show (between seasons 2 and 3.) It had all of the familiar characters with their respective actors doing the voices, the same composer, and even the same writers.
But the controls are a bit wonky and the driving missions just plain suck.

Something like this came up recently for JRPGs specifically, so I'll link to my post there to save space
http://www.gog.com/forum/general/looking_for_a_jrpg_on_gog/post52

I play very little that isn't an RPG of some description, but to add to that:

Legacy of Kain: Soul Reaver 2 and LOK: Defiance, if for some reason you don't wan't them from GOG. LOK: Blood Omen 2 is a bit on the sketchy side, but might be worth a look as well. Also look into the PS1 predecessors, Soul Reaver 1 and Blood Omen 1. The original Blood Omen in particular would be a good find, as it doesn't seem to be on GOG for whatever reason.

God of War is worth looking at. The first one was pretty good, but heavily marred imo by some incredibly frustrating tightrope-walking sections. The second was much better overall.

Zone of the Enders (1&2) are pretty fun. Sort of fast-paced arcadey mech shooter games I guess?

Front Mission always looked good to me but I haven't personally made the time to try them out. Can't call that a recommendation.

Last games I can mention are the Devil May Cry games. Now, personally I didn't like them much at all. But they are pretty well respected action/beat-em-up type games, so if that's something you enjoy take a look.

Definitely play Zone of the Enders 1+2 if you like mech-type action games - Oni is a pretty good action game from Bungie too.

To name a few...

Gun
Okami
Disgaea
God of War
Summoner
Odin Sphere
The Bouncer
Devil May Cry
Resident Evil 4
Final Fantasy X
Unlimited Saga
Kingdom Hearts
Xenosaga series
Onimusha series
Metal Gear series
Jak and Daxter series
Star Wars: Battlefront II
Ratchet & Clank series
Shadow of the Colossus
Beat Down: Fists of Vengeance
Psi-Ops: The Mindgate Conspiracy
Prince of Persia: The Sands of Time
Dragon Quest VIII: Journey of the Cursed King

Gah...I had written up a huge post yesterday and then the forum glitched and it disappeared.

Suffice it to say, like you, I got a PS2 earlier this year, and it was a great decision. I've never been much of a technology hound, and I'm certainly not going to buy a new console the moment it comes out (and with the current gen, considering the stupidities, I may never get them).
Ok, so I'm usually not a fan of modern JRPGs, and I don't get horror games, and I'm a bit inexperienced with modern controllers (last one I used was the Sega Megadrive II one): just so you know my biases.

People are going to recommend you a LOT of 3D platformers, and that makes sense, there are a lot of great 3D platformers on the PS2. Most of them have a lot of overlapping experiences, so you probably don't want to get ALL of them. Highly recommended are the Jak & Daxter games, Ratchet & Clank, Sly Cooper 2 (Apparently 3 was just average, and 2 was better than 1) and Viewtiful Joe.

Another game that EVERYONE (even here :D) will recommend to you is Ico. Great game, not much more you can say about that. Shadow of the Colossus is a sort of spiritual sequel (prequel?), and will also be highly recommended. Personally, I preferred the simplicity of Ico, but SotC is good too.

As I said earlier, I'm not a fan of horror, but Silent Hill 2 is another game that will come universally recommended despite that. Metal Gear Solid 3 and Metal Gear Solid 2 also both come highly recommended, but I'm still on 2, so I can't say about 3 yet. 2 so far is great, even if it is a bit frustrating at times ("Snake? Snaake?! SNAAAAAAAAAAAAAAAAKE!" :D) I keep on with it. Okami is another great one. Red Dead Revolver was hugely surpassed by its much better sequel on the PS3, but it is still a nice little game (although I'm still not used to the targeting, but that is probably just me). I was also recommended (and got) Shadow of Destiny/Memories, but haven't had a chance to play it yet, so I can't speak to its quality.

I was recommended Final Fantasy X and Final Fantasy XII, so I got Final Fantasy X, and I had enjoyed the SNES FF games (3 and 4 and 5?) and even played VII on a PS1 emulator way back in the day. Perhaps I'm not the intended audience, but I got bored with FFX relatively fast. It had a 30 minute long intro that seemed totally irrelevant (a space-football player revving up for a game when suddenly the world ends), followed by a corridor of boring clickfest battles (I don't mind turn-based battles, but right at the start it was jut "Click attack, attack, get attacked, click attack again, repeat until one is dead"). If you like those sort of games (huge over-acted cutscenes with minimal linear gameplay inbetween), you might want to get those.

Now just to show I'm not a bigoted japanophobe, I was also recommended Way of the Samurai, which I found great. No boring huge uninteractive cutscenes, but nice non-linear game with emphasis on NPC interactions (which as an adventure gamer, I really enjoyed), and some cool sword-fighting mechanics. Way of the Samurai 2 was also recommended to me, but I didn't have a chance to get that, so I can't speak on it.

Some people might say they're childish, but the Katamari games are awesome fun. Apparently We ♥ Katamari is better, and I was told that I should play Katamari Damacy to appreciate 2 more, but I don't think it was necessary. I just have We ♥ Katamari, and it is great.

For PS2 game experiences, the God of War games are a necessity. I got the first two, and played II first, being told it was the better game, but it was probably focused more on a more experienced PS2 player than me (it had QTEs with sneakily changing buttons, so by the time it said "SMASH X TO LIVE" and I looked down at my hands to see which one was X, press it and look back up, it was too late), and 1 was much gentler.

For me, I didn't get any games that I had already played on the PC, but if you want those, you haven't (or if you want to replay them) you should get Prince of Persia: Sands of Time, Beyond Good and Evil, Psychonauts, and the GTA games. If you enjoyed the GTA games, Liberty City Stories and Vice City Stories never came on the PC, so you might like those.

For the token fighting game, you can get Tekken Tag Tournament. For the sports game (don't laugh, it is great! :D), you can get SSX Tricky, or if you can't find that, one of the Tony Hawk games. For rhythm games, if you can find it, get Gitaroo Man, or if you can't find that, Rez or Frequency or Space Channel 5 2. Rumble Racing is the Mario Kart of the PS2, and other cool racing games are Micro Machines, Gran Turismo and Burnout.

Other games I was recommended highly, but simply couldn't find (or they were prohibitively expensive): Time Splitters 2, Kingdom Hearts and Odin Sphere.

And the rest of the recommendations to me, in decreasing order of intensity (which either through not being able to find them cheap, or not being a survival horror fan or super-japanophile, I ddin't get):
Haunting Ground, GunGrave, SOS: The Final Escape, Clock Tower 3, Spy Fiction, Silent Hill 3 and 4, Gungrave Overdose, Shinobido: Way of the Ninja, Glass Rose, Gregory Horror Show, Project Zero/Fatal Frame 1, 2, and 3, Persona 3 and 4, Devil May Cry 3, Dragon Quest 8, Dark Chronicles, Rogue galaxy, Disgaea and Disgaea 2, Makai Kingdom, Ring of Red, Steambot Chronicles, Mad Maestro, Freak Out / Stretch Panic, Chaos Legion, Mark of Kri, God Hand, Maximo, Zone of the Enders 2, Onimusha 3: Demon Siege, Dragonball Z Budokai 3.

Whew...that turned out just as long as before. Hope it was helpful!

Most games I'd have recommended have already been mentioned, but The Hobbit hasn't. Not an extraordinary platformer by any means, but still good fun. And even if this might provoke some sneers from the classic RPG purists, if you're looking for a hack and slash coop that's easy to pick up, you could also give Baldur's Gate: Dark Alliance and Fallout: Brotherhood of Steel a try. Or, if you like a brutal challenge, Alien Hominid by the Castle Crashers devs.

Thanks for your effort of writing such a huge post I think I will buy Metal Gear Solid 2 or Devil May Cry 1(found them very cheap)

Is Resident Evil Code: Veronica X any good ?

Nobody's mentioned Spider-Man 2 yet? Before Batman: Arkham Asylum arrived this was pretty much the best superhero action game there was. An entire city to swing around and explore, side missions everywhere and generally just a heap of fun.

Also this may be controversial but Escape From Monkey Island is pretty damn good too, though I'm sure many purists of the series would probably disagree.

Finally, of course, the PS2 is backwards compatible so you have the whole stunning lineup of original Ps1 games to choose from as well.

I'm glad it was useful, voslobandre!

Lies! Slander! Filth!
I dunno, I guess I'm one of those people this'd be controversial for :P. I played it on the PC, though, and I found the game had rubbish graphics (they used the same character model, without even changing colours, for about half your crew!), the gameplay was irritating (one word: BLEKH. I mean, monkey combat...) , and the story was boring and childish (especially compared to Monkey Island 1 and 2).

If you like Horror games Fatal Frame is an exelent option