Whats the best resolution to play the game at? I had it at 1080p but its just too pixelated.
And 800x600 is too small. Is there a middleground that people like to play these games at?

There is no 'best' resolution, only what you find acceptable. Since you mention (inaccurately) that you were playing at a 16:9 resolution (most likely 1920x1080), you must have Bigg's Widescreen mod installed. To that end, you should probably go for something along the lines of 1280 x 720 (for the same aspect ratio), or 1280 x 800. For a more standard, yet lower, resolution you could try 1024 x 768, though this is more of a 'square monitor' resolution. Experiment until you get something acceptable to you.