I hope to play these games in windowed mode, with the guide open (journal references, maps, etc), but every time I tab out of the game to desktop, I'm left with a tiny window to play the game in, and cannot seem to resize the dosbox window. How can I do this?

Alt-Enter

Yes, I do that, but the box with the game in it is very tiny afterwards, and I cannot seem to make it larger.

Try different scalers (3xnormal, for example) and renders (ddraw, surface, overaly), and the fullscreen and aspect ratio options from the Graphic options.

"The Graphics options are located via a separate application that GOG provides. It should show up under the same start menu grouping as the game itself (or you can find it in the game's folder). It's a small little front end that adjusts the graphics options in the DOSBox configuration file for you, so you don't have to root around with Notepad yourself."

Thanks for the tip! 3x normal did make it bigger. I'll keep looking through the other ones, but this alone makes it far more playable.

Can you please explain in full detail step by step how to actually do this? I can't find the right file! Let alone the setting.

I don't have the GOG versions installed.
And I use D-Fend Reloaded as a front end.

But the separate application mentioned should be called Launch Graphic Mode Setup.

Or you can edit the .CONF file manually.
Look for this section:

[render]
# frameskip -- How many frames DOSBox skips before drawing one.
# aspect -- Do aspect correction, if your output method doesn't support scaling this can slow things down!.
# scaler -- Scaler used to enlarge/enhance low resolution modes.
# Supported are none,normal2x,normal3x,advmame2x,advmame3x,hq2x,hq3x,
# 2xsai,super2xsai,supereagle,advinterp2x,advinterp3x,
# tv2x,tv3x,rgb2x,rgb3x,scan2x,scan3x.
# If forced is appended (like scaler=hq2x forced), the scaler will be used
# even if the result might not be desired.

frameskip=0
aspect=true
scaler=normal2x

The last line is the one you want to edit.