I bought underworld 1 and 2 years ago but never really got into them. So i'm going to try to play them again but Now I'm on windows 10 and the game window is really tiny surrounded by massive black boxes something that didn't happen years ago. I've tried playing with the graphics options in the dosbox configuration tool and while I have managed to make the game window larger it's still tiny and unplayable. Is there a fix for this?

have you tried the scalers is dosbox.conf?

I'm not sure what you mean or how to do that.

Go into the directory, that you installed Ultima Underworld into (on my computer it would be D:\Games\GOG\Ultima Underworld) and start the "Dosbox Configurator.lnk" (picture 1).

Pick the Basic Settings (picture 2) and the Advanced Settings (picture 3), click on "Save Settings" and then on "Start Game".

The game should then run in a nice big window. If you don't like what you get, experiment with different settings.

It works the same for Ultima Underworld 2 and any other GOG game, that runs in DOSBox.

No that's my issue none of the settings cause it to go fullscreen, the actual game window (where the UI is) only takes up at max like 30% of my screen.

What happens when you play with different Scaling Engines and Window Resolutions in the Advanced Settings tab?

It changes around the window size but only by a little. I haven't tried every option so I can do that first but I'm not hopeful about it.

Edit: Well here's something interesting. I accidentally windowed my game and the window was at the max resolution (1920 x 1200) that the tool allows but when going back into fullscreen it's back to the small window. So i'ts playable for me in windowed mode but still not ideal.

I've just been through this myself, the changes I made to the standard dosbox conf file to fill the height of a full screen were:
changes under [sdl]
fullscreen=true
fullresolution=1920x1080
output=ddraw

changes under [render]
aspect=true
scaler=hq3x

Can confirm these settings work in 4K resolution without issues.