First off, i was playing a game last night and my screen went all corrupt. I figured it just overheated so i shut her down and moved on, later when i went back it ran fine no glitching until i got to a massive area in the game with a crap load of shaders. it then froze and kicked the bucket, no game will load or it does and then goes all wonky, color blocks everywhere.
Sadly my 8800GT workhorse has died. And i need a new card, these are the 4 cards I'm looking at: HD5770, GTX260, 9800GT, HD4870 -- my last 2 Nvidia based cards both had either bad shader cores or bad memory so im a little on fence about getting another. Thankfully windows runs fine, just not games...
The HD5770 is what is looking good to me, Runs Cooler with better power usage of them all. But its lacking in performance where others exceed. Keep in mind My target resolution is 1680x1050. I will not be going higher than that.

The HD5770 is the only DX11 card of the bunch, so if you for some reason are looking for that, the Radeon 5000 series is your only choice so far.
Other than that... well, I like AMD better than I do nVidia, so I'd probably get that one anyway, preferably from Sapphire. I have an HD5750, running 2 20" displays at 1600x1200 each.

Ive never even seen DX10 so Dx11 is nothing, besides If my eyes still think a Dx9 game looks beautiful (Fallout3 Crysis, Risen, UT3, Hellgate) then when i see Dx10 im going to shit a brick, and maybe half a HOUSE for DX11 :)
But i digress, I have been burned by Nvidia to many times, i think i might go ATI. Unless someone here can make me think that Nvidia is still viable, also Hydavision and Multi Monitor, SLI, Phys-X doesn't concern me. i usually dont run any AA or AF. Just V-Sync, shaders, and HDR... Fallout 3 is the only game i used AA for since Nvidia hates there transparency Multisampling.

The difference between DX10 and DX11 is that people are actually using DX11. Unlike the so called DX10 games which put a DX10 overlay over DX9 which made performance suffer. The 5770 is your best bet out of the bunch.
Out of the other options a 4870 1GB will wipe the floor with the other cards you mentioned.

Yep, and effectively removing one of the best improvements in DX10; not having to copy texture data into conventional RAM.

My only issue is that the 5770 is the new kid on the block and doesn't have much power to run DX11 games. Even at medium Crysis Warhead was benchmarked around 34...
I'm using a 7900 GS right now as backup to keep my machine alive, so anything is better.
This is what i found, if there was another DX11 card around this price point $150-200 then i would jump on that, but for bang for buck this sounds like it. SAPPHIRE 100283L Radeon HD 5770
Well CRAP that 7900GS is causing all sorts of Havok in my system... gotta strip the drivers and reinstall... ugh and another memory address BSOD due to the GPU.. now i remember why i replaced it...

Right now, in terms of hardware, ATI's 5000 series completely outstrips everything else. But their continued incompetence with drivers puts me off them.
Definitely avoid Nvidia's 9000 series as it was nothing more than a re-branding of the 8000 series.
Personally I opted for the GTX260 which can max out most games (Crysis included) comfortably enough while benefiting from solid drivers that don't seem to get worse with each new release like ATI's.
I'd also recommend as a general principal you opt to upgrade more frequently using cheaper cards than upgrade to the best. As, by the time the best card is in any way taxed by games it's a year old and a fraction of the price you first paid for it. Pointless.

I'm tempted to go with nvidia, I mean "I know them" I know there drivers... but on the other hand i have a laptop with an ATi card, most compatible thing Ive ever seen. never even updated the drivers and its playing games 2 years past its production date fine... no heat issues no glitching, just a good solid card...
So hard of a choice.... wont be able to upgrade for atleast 1-2 years again

You really don't want to go AMD graphics with an Intel CPU : That, with bad drivers will reallllly cause problems. If you've not got an Intel CPU, then AMD will probably be ideal....
*walks away trying not to laugh*
Seriously : Stay away from AMD/ATI cards if you can.

Before you go nuts, have you thought about pulling the heatsink off, giving it a good clean and putting some decent new thermal paste on? There's a possibility that it could just have lost almost all contact with the heatsink and is overheating extremely quickly. If nothing else, its not like you could break it any more than it is...
I'd definitely go with nvidia as a replacement though, I've never had a good experience with ATI cards

Kind of depends what games you see yourself playing in the next year or so. If Fallout 3 is going to be the most graphically intense game then the 9800gt will do you fine and its around the $100 mark so you can buy something better in a year. The HD 5770 is the most powerful, but it is a first generation dx11 card so how well it handles future dx11 games is yet to be seen. I'm also not a fan of ati drivers, have nothing but trouble with them.
I'd probably go with the gtx 260 myself just make sure to get it from a good company. I normally go with BFG because of the good components, customer support(real people that speak fluent english) and they also have 10years warranty on their cards.

If you shop around you can pick up GTX260's at bargain prices ATM which generally makes them better value than the 5770, the other consideration is that the nvidia line is built on tried and tested tech(some might argue old) as a result are likely to have much more stable drivers.
In the end I would just find the best bang for the buck card you can find regardless of recommendations.

How is the rest of your rig? State of the art? Likely to get most components replaced sooner rather than later? If it's the latter, go for the cheapest alternative that will cover your needs. If it's the former, think future-proofing.

That's the problem. They routinely release drivers on a monthly basis. Nvidia release them as and when needed, as and when ready.