It could be a number of things as scaling is done via your GPU drivers, some monitors do it too, then there is DOSBox aspect ratio and lastly we have nGlide. All of these can do scaling, so any of them could be causing conflicts with others.

For me, I have my GPU used for scaling as my monitor doesn't scale it stretches. So in my nvidia Control Panel I've set it for GPU scaling. Then with this game I have:

fullresolution=original
output=direct3d
aspect=false

So with these settings you'd expect my GPU to take the original game resolution and scale it to my monitor thereby keeping a 4:3 resolution.

BUT

As I'm using the 3DFX version, nGlide takes over, and my nGlide settings are:

Screen Resolution : By desktop
Aspect Ratio : 4:3

And therefore when in the 3D part of the game it plays in 1680x1050 (scaled by nGlide) but with pillarboxing to keep a 4:3 aspect ratio.

So the DOSBox settings don't seem to be used when in the 3D parts of the game as nGlide takes over, but in the menus where it is all 2D the DOSBox settings do appear to be used, and hence why it is all scaled correctly.

Yeah, same here with the monitor and gpu scaling. Just very strange as this is the first and only DOSBox and 3Dfx game i've had a scaling problem with. Well, except for Tomb Raider 1's 3Dfx mode, but that was solved by simply deleting a glide dll file as with Carmageddon's 3Dfx mode before gog updated the installer for it.

I finally found the reason for the stuttering in the 3dfx version:

https://www.gog.com/forum/archimedean_dynasty/stuttering_soundmusic_and_falter_gameplay/post9

(Turn of vsync in nglide settings and nvidia control panel.)