I've had a consistent issue in FS2 in Mission 10: Mystery of the Trinity. It's the first mission set in a nebula, which I think may be related to this problem. At a random point in the mission after discovering the capital ship in the nebula, the graphics start getting a bit choppy. After a few seconds of this, most of the HUD disappears for a moment, and then the entire screen turns bright yellow. Pressing hotkeys still causes things to happen in the background, but I can't see it and they're delayed several seconds. Pressing Esc, then tabbing out and back in, brings me to a black screen with the normal resume/quit/restart dialogue options. Resuming brings me back to the game for a couple seconds the first time, and then the cycle starts over. Trying this any additional times only gets me a blank black screen, and the game crashes shortly thereafter. I've tried tweaking graphics settings, since that's where I think the problem would be, but to no avail.
I'm running Windows XP-32x on a 2.4Ghz 64x AMD Athelon, GeForce 8800 GT graphics, 2GB RAM.
Any ideas?

Are you using Freespace 2 Open?

I don't think so? Not sure what that is. If it's a mod, though, I haven't done any of that - I'm using the raw version downloaded from GOG.

You might want to check it out (Here is the Wikipedia article on it: FreeSpace 2 Source Code Project ). It isn't a mod as much as an updated engine using the released source code. Installing it would be the first thing I would do if you can't run it as is.
Check this discussion out: Freespace source code project.
And this one will help you through the installation: http://www.gog.com/en/forum/Freespace_Series/gog_freespace_2_freespaceopen/perm=3/#p_b_3]
Q: GOG Freespace 2 + FreeSpaceOpen?[/url] (I used the automated installer they mention as an alternative).
Check it out!