For me, the times when I felt the most deliciously evil in games (instead of just stupidly evil as you slaughter innocents for no reason) were mission endings in Dungeon Keeper and the evil ending in KotOR.

There are plenty of RPGs where you have the choice of playing as an evil bastard and the game won't punish you for it, but I think KotOR was the best at actually enabling it, and the ending is very satisfying (much more than the good one imo).

Lots of games have already been mentioned, though in some of them you play more of an anti-hero than a real villain. Some more not mentioned:

Contract J.A.C.K. - a spin-off to the NOLF series and a pretty poor one.
Emily Enough: Imprisoned - a free point'n'click game and a bloody hilarious one.
Stubbs the Zombie in Rebel Without a Pulse - braaaains
Dark Skies: The Heritage - you play as a son of an evil overlord that has recently died. A hilarious parody of villainous characters. It's a shame that only a demo was released and that it appears to be completely gone from the internet.
Dooms saga (including Dooms, Dooms 2: Seas of Blood, Dooms 3: Rough Day and Dooms 4) - the game tries to be as offensive as possible by giving us bloody murder, rape and cake... It's also completely free.
Saya no Uta - the evilest game that I have ever played :-) There are not many things more evil that helping a Cthulhu abomination to take over the world.
Downfall - a creepy horror game by Harvester Games.
Higurashi series - in some of the chapters you play from a villainous perspective.

Why do I have the feeling your catalogue of games consist only of "Postal" "Postal 2" & "Sacrifice" ???

Duke Nukem Forever

HAHA no. Oh, and thanks for reminding me. Sacrifice is pretty evil as well if you play with Pyro or Charnel, you capture slaves, slaughter people,steal souls, etc.

Call of Duty: Modern Warfare obviously. You participate in a global genocide, often in a squad lead by a British sadist who enjoys torturing people. All in the name of the new world order!
I think he meant games with evil content, not games that are an evil joke themselves.

Ah, so that's the title of the game. I only heard of it, first time I see it. On a conference about the infamous German rating agency USK some guy took this as an example why PEGI is also quite a fucked up system. The people doing the ratings just follow a checklist - in this game you apparently get to force another character into suicide by terrorizing him psychologically and since there's no such thing on the PEGI's list the game got a 12+ rating which even the developers themselves found disturbing. Gotta love the ratings.

Always wanted to play that game, but I don't have a console, and none of my friends who have one have the game as well :(

Feeling guilty is easier when characters are a bit more fleshed out than spacde invaders flying saucers or carmageddon grannies. Maybe Project Zombies gives some opportunities for that. Also, a few years ago, I was showing Tropico to a friend, and describing the various functions. He absolutely wanted to see assassination, so, I targeted a political rival, and sent my militia after her. We had first to locate her in the crowd, and then to trigger the assassination, which too time to get carried out. In the meantime, I showed my friend how we can follow that person in the crowd, in its daily activities, how we can read stats and info on her life, her job, activities, and her thoughts, her goals, etc. As we were following her, and getting details on her, she was getting more real to us, we were progressively getting a bit ill at ease. Then a bunch of little virtual guys in uniforms arrived, fired a submachine gun, and left, and she was dead. After one or two seconds of silence, we decided to play something else for a while.

We were totally depressed for the whole evening.

Reminds me of this moment with the pencil example :

http://www.youtube.com/watch?v=iE8meUf0qCM

I've noticed that there are 10 types of people, those that use The Sims to torture people and those that have never played The Sims.
I'm pretty sure that Postal 3 is out, but I haven't actually had a chance to play it. The main problem with Postal is that the controls are a bit wonky.

But yes, both evil and grim. Postal 2 is evil and somewhat less grim. But definitely evil.

OTOH, who amongst us can say that they never considered gunning down the entire marching band in high school?

Me. But it may have something to do with the fact that I don't know a single German high school that has a marching band.

Postal 3 has been out for a while. I played a couple of hours in a net cafe and I was... Pretty disappointed :( It has no humor, the 3rd person perspective is horrible and worst of all, the bastards got rid of Rick Hunter :((