I can't find anything on how to do this, except one GoG post that says to just hit alt+enter. ..Which does not work for me.
I'm on windows 10, and I alt-tab allot, so it'd just be easier to run it in a window, but now?

I'm not sure about the unmodded game or Co8 only. If you use temple+, run the config program. there you have the option to set windowed mode.

Without mod someone mentions add -window to command line in the windows shortcut:
https://www.gog.com/forum/temple_of_elemental_evil/windowed_mode/post2