I've never thought of monsters as scary. Just grotesque.
When I play a horror game, it's the tension, atmosphere, and foreshadowing that I look for in the experience. I might add, I don't think any other game I've played has managed to perfect these three aspects as well as Penumbra: Overture and Black Plague, and maybe Cryostasis.

The fact is: monsters are as scary as ever, but nowadays games are shit in proposing them to the player.
So blame games developers and not just monsters themselves. They are victim of their creators dumbness :-P
(oh, and you should definitely read H.P. Lovecraft if you haven't done so)

My idea for a good horror game: One that changes everytime.
Randomly generated campaigns, a large pool of monsters strategically placed throughout the level by an AI director, (a la Left 4 Dead), the AI Director controls the lighting and events.
Game randomly shifts from normal reality to horror reality. (Longer and harder each time)
(Sort of like in Silent Hill)
Maybe have a Phobia system, where the more the character you selected has, the harder the game becomes, (so the game would have more emphasis on timing and stealth, possibly to the point that that character is practically helpless, and can have panic attacks, (sort of like in Penumbra)).
But also the Phobias reflect the level generated as well.
A character with claustrophobia may have a level with lots of narrow corridors and small rooms with important items. A player that doesn't want to take a risk may find a way arround.
HOWEVER, in true horror style, the AI Director may decide that you are playing it too safe, and might make the game even harder later on, (with a quite sharp difficulty spike), so the game may play on people's knowledge, ("oh, a monster is obviously going to jump on me if I go into that room, I guess I better not go in that obvious trapand take the route round" 5 mins later: "hmm thats odd, no monster have jumped me" HEARS NOISE- Cue mouse 20 mins later Leave house and walk down path talk to neighbour- OMG MONSTER STRUGGLE,. Wake up in dank cellar, do puzzle to get key. Move through gory version of spooky house you were just in, other side of room you didn't go through
-try to move around, Door locked, (may not be locked if you went through room earlier)
- go into room, hear monster approaching room,
- hide! monster gets closer and closer, spots you. Wake up in hospital. Nurse says you collapsed and neighbour called an ambulance
or
-don't hide, monster "slaughters" you. Wake up talking to neighbour. He though you spaced out for a moment. You shrug and head back home, realised you forgot to buy present for daughter, and head to supermarket, (which would be easier to "escape" from than the hospital)
So the game can punish you for playing too safe, but may also punish you for being too bold as well, (with the phobia system).
(Sorry if this is a mess, but I thought I'd type as I think, (i forget my ideas almost instantaniously))

I loved Call of Cthulhu: Dark Corners of the Earth. That game scared me real good. There was one part where I was more panicked and scared than I had ever been in a game before (and since), and there weren't even any monsters at that part.

Scared of gaming monsters? Nah, though I can still get? startled. But I am not a kid any more, or even a teen. It might be natural to be less scared the older you get, I don't know, this is my only lifetime that I know of.
I remember getting scared playing the original Test Drive, first on a PC, CGA graphics and then on my Atari ST. Going 160 miles an hour with a cliff on your left side, rocks on your right, traffic on mostly two lane roads, police in the back and all that with single digit frame rate...how can a monster scare me after that experience? After seeing a station wagon coming at you (or you at it) with over 200 mph and be able to see their faces as you kill them and die too...now that is scary!
The other thing is, music. It is just not well done in most games. It is usually so bad you turn it off. Music should be there to set the mood, not to irritate you or to let you know you are about to meet something hostile.
The other thing is, there is no sense of awe in games that need it, like fantasy or sci fi based games. (with very few exceptions).
There is also the too much info problem, or consolitis. When you can't clearly see the enemy if at all, yet if you move the cursor/etc it brights up like a xmas tree when it is above hostiles even when you can't see them due to distance etc.
There there are games like FEAR 1. You see the girl or other "ghosts" and you get scared the first time or two. Then you realise they can't really harm your character till near the end of the game and you are like "why can't I skip this?"
Repetition is another problem. When you know that "scary thing A" is going to happen when you reach point B, you stop worrying or of course getting scared by it. There is a need for randomness in games.
The last thing is attachment to your character(s). You need to feel close to them and that they are very fragile. Case in point X-Com, especially the night missions. (plus the music)

Children do that to me anyway, they don't have to stop and stare >.<

The problem today I think its that the "scarier" the monster the better weapons you have. So you just dont feel threatened. More like how am I going to kill it using my BFG9000.

True.
Also true.
About Lovecraft's stories: while his settings are debatable with regards to believability (mad symmetry, ancient intelligent alien lifeforms), and the nemeses sound downright cheesy at times, you have to hand it to the Master of Horror for being able to make a story sound absolutely terrifying regardless of what or who is involved.
Also, if you like Lovecraft, may I also recommend his good friend Robert E Howard aka creator of Conan. Both of them have settings that are based on almost identical cosmology mythos. I believe the unanimity is due to the consistent correspondence between both men, at least when Howard hadn't kicked the bucket yet.

I'll put in another word for Amnesia: The Dark Descent. I'm a long time fan of survival horror games. I even loved Haunted House on the Atari 2600 when I was kid. I've played a lot of the major series of horror games over the years- Alone in the Dark, Resident Evil, Clock Tower, Silent Hill, Siren, Fatal Frame, Penumbra, etc. Nothing nails it quite like Amnesia: The Dark Descent.
Why does it work? Because they understand fear. Fear is about avoiding danger and approaching safety. The game teaches you to avoid the dark, to fear the dark (you have to keep your character out of the dark or he gets scared and starts to lose his sanity). But when a monster comes around, you are forced to hide in the dark. You instantly understand that the darkness is bad, but the monster is worse. You are also weaponless and fairly helpless. You have to run way and hide. There is no other option. I don't even know what many of the monsters look like in that game because I've only seen glimpses of them when hiding or running away (while yelling "oh shit oh shit oh shit" in my PC room).
Another thing Amnesia does well, is it models interoceptive experiences. This aids in immersion. They are not the first to do this, for example in Mirror's Edge, when you walk out of the dark and into sunlight, your screen is whited out to give you the perception of needing your eyes to dilate and get accustomed to the light, or in Half Life 2 when you get hit with an explosive you lose your hearing and are temporarily left with the squeeling tone of tinitus. In Amnesia, you have similar private interoceptive experiences of the character modeled in the visuals and sound. When you first walk way from a torch, everything is black and it takes a moment to see in the dark again. When your character panics, you hear his heart beating in your ears and the screen blurs as if he is in a panic. The mind plays tricks on him and he starts to see bugs crawling that aren't there when you look again, or you hear scratches and creeks that are only in your mind. You hear his breathing, his gasps, his screams. They basically mess around with your perception to cue you into being scared and to make you question the character's very sanity.

Playing Doom 3 right now. It has its scary moments, but that's all the atmosphere, no the actual monsters themselves.

That's why monsters in NetHack can be some of the scariest of all, even though they're just ASCII characters, and it isn't a horror game....

What new games need is a more emotional attachment with the games at the very start and throwing you in a situation where you are unprepared, not knowing what will happen and what is beyond you knowledge. The setting and environment is most important to give you the right shivers.
So far only Frictional Games managed to perfect the horror genre, and knew how to use the first person gameplay as a new form of horror adventure.
Now, you don't really need to be a helpless little twat at each game and can still be very well prepared, but the trick is to make you operate in a surreal environment where everything can happen and throw you off guard, and you are kinda creeped out what horror you will face to survive.
Best example would be Super Metroid Prime on the SNES. I know, it's not a horror game, but regarding environment and setting it's very close. This is still one of the scariest moments on the Super Nintendo:
http://www.youtube.com/watch?v=ONxKm8uApSc

Last game that scared me was probably GTA4, but that was from being hit by cars at random, not from monsters.

To be fair, Shock 2 has its moments as well *shudders at Engineering deck*. Again, hybrids are creepy, but the greater fear in Shock 2 lies not with the hybrids, but with the damned protocol droids and their big brother maintenance droids. Normally, I'd scoff at you if you told me that a droid that walks up to you and says "Can I help you Sir" is pissing scary, but Shock 2 proves otherwise. Of course, it becomes less scary later on, and probably a little frustrating (Cyborg ninjas), but nonetheless, the early stages make you feel helpless and really do a good job at making your skin crawl. It's the same thing with LGS' other game - Thief. Both Thief and Shock 2 have atmospheres that are perfect at times.