As the subject line says, I'd like to play the game in 512x384 without having a huge black area around the actual game display.
I'm running Windows XP SP3, my GPU is Nvidia GTX 260 with the 197.45 drivers, and I'm using the "Nvidia scaling" option from the drivers. I have tried it with my monitor's scaling as well, it makes no difference. (Other specs, should they make any difference at all: Intel Core 2 Duo 3.0GHz, 4GB 800MHz DD2, Sound Blaster X-Fi XtremeGamer.)
Basically, I can remove the tick from the Framed option box, start the game, notice the huge black area, quit, go the options to find out that the Framed-box is ticked again. What gives?
Also, I have pointed Windows to the correct .inf for my 1920x1200 monitor, so all resolutions ranging from 800x600 - 1920x1200 are listed as usual.

As far as I understand it, the resolution of 512x384 isn't supported by most modern monitors or graphics cards. Thus, whenever you disable the "Framed" option, Outcast reenables it when it fails to run in 512x384.
To use that resolution, you have to manually add it as a "custom resolution" in the driver software of your Nvidia graphics card. How you can do that is explained at the very last passage of this post: http://www.gog.com/en/forum/outcast/changing_to_a_viable_resolution/perm=57/#p_b_57
This worked well for me, I was able to play Outcast in fullscreen. I've only encountered the problem, that sometimes after quitting Outcast, the desktop resolution would look a bit distorted, as if the change from 512x384 back to the desktop resolution didn't go so well. But, simply changing to another resolution and back, or briefly starting Outcast and quitting it again solved this every time.

Thank you for your help, Syrion
Edit: Well, it didn't work though. Also, I tried most of the tricks from
http://www.gog.com/en/forum/outcast/changing_to_a_viable_resolution
but nothing works. The game window is always ridiculously small. It's probably time to use one of those widescreen patches.

I would accept the explanation that my monitor doesn't support 512x384 resolution UNLESS I had 100 % fullscreen once but then lost it for some totally unknown reason.
I have the same problem as you mrfz, the 'framed' option reticks itself, and if I remove write permissions from the outcast.ini file (or was it loader.ini?) the game simply refuses to start.

You could always give this a try:
http://www.gog.com/en/forum/outcast/final_640x480_fullscreen_patch_stable_some_aesthetics_problems_fixed