In Windows 10 alt tabbing back into world of xeen reopens the game in a small window that cannot be maximized. Is there a way to tab in and out of the game?

Play it in a window?

By configuring DOSBox, either manually, via the Graphic Options program which I think the GOG version includes, or using a front end like D-Fend Reloaded.
Or you can change it on the fly by pressing Alt-Enter.

You should be able to maximise the window again by pressing alt-enter. You might have to click in the dosbox window a time or two to make sure it is in focus for it to work.