Well, there is one thought I'm having but it's kind of an extreme solution.

And that's to install Linux, get Wine installed, and see about running it there.

Yeah, go figure.

I wouldn't be surprised if that worked. ;)

Installed and launched through GOG Galaxy?

GeForce RTX 2070
AMD Ryzen 7 2700X Eight-Core Processor

I'll attach one of the problem reports windows gave me.

edit: Oh, and I tried most of the settings in 3DSetup that opens through GOG settings / the game folder. Removing "srDD_Glide2x.dll" in the dll folder gives me a whole lot more resolution options there, but it doesn't help me launch it either.

EDIT: Yes. I installed the game through Galaxy.

I know this may sound stupid, but I wonder if it's something related to the graphics card. I've heard people with RTX hardware complaining of issues trying to run older games. Since Wiz8 was one of those games optimized for early GPUs (3DFX) I wonder if there's a graphical incompatibility to blame.

Graphics card is very likely culprit here, especially because you have an RTX card. Most likely its drivers aren't compatible with W8, as the current priorities with the card lie elsewhere than trying to be compatible with old games.

I ask this because you didn't state to have already tried it, but have you tried changing your desktop resolution to 800x600
and see what happens?

It may well be my graphics card.
Changing the resolution also didn't work but thanks for all the suggestions!

There is a revised unnoficial opengl driver/patch for wiz8 out there, a "srDD_OpenGL.dll". It served to solve some texture glitches ingame. But who knows. Maybe it is more compatible with some modern cards combos. It is worth a try.

http://www.zimlab.com/wizardry/recovered/wolfie/Wizardry8BUG-FIXES.html

EDIT: Select your desired renderer in the config program, but alternatively edit these options manually in the 3DVideo.CFG to be in the safe side:

OpenGL
1024
768
32
Miles Fast 2D Positional Audio

Sadly didn't respond to these changes either.

Following the suspicions about problems with RTX cards, I came across dgVoodoo and tried fiddling with that.
https://www.gog.com/forum/wizardry_series/wizardry_8_cutscenes_and_widescreen_solutions
For some people it helped with running old games. I'm not very talented with hardware and software so it might be that, or it requires some obscure combination of the brought up suggestions, but Wizardry remains unresponsive.

You can try and ask for Support to help you. If you bought the game recently and if they can't help you, they may probably refund you.

https://support.gog.com/hc/en-us/articles/213038925-Wizardry-8-does-not-launch?product=galaxy

This is the link for the app that apparently fixes the "unable to launch from gog" problem. The pcgamingwiki one seems to be broken

EDIT: And this is the direct link to the file itself.

http://files.gog.com/support/wiz8_fix.zip

We tried that as admin but all it does is tell me the game isn't installed despite fresh reinstalls and repairs etc. I think the fix is expecting an outdated filepath. Still thanks.
Since it might be my hardware I'll try to reach out to support for help or a refund.

So just coming back to this. Your not unfortunately entitled to support on this. It clearly does not state compatible on windows 10. It also seems most of the usual has been tried. Can you try it on another machine, maybe a win 7 device? An old laptop or desktop with win 7 on it, just download the offline installers (do not download or install anything galaxy]. Install from those to a folder outside windows folders, c:/mygames should be fine. Again I say do NOT install to windows folders, and do NOT install galaxy. If it still fails to work on a win 7 machine with those two criteria, then you have a valid case for a refund. Running through galaxy on an rtx20xxwin 10 build isn’t.

Not that this will help you, but for me Wiz8 GOG version works fine out of the box, including the videos. Windows 10, GPU is NVidia Quadro P600 (a laptop).

I used the offline installer for the game, I don't have Galaxy installed.

Even without refund they might be able to offer some insight. I will try an offline install later as an last attempt before taking a break since this already took several days of research.
I might be able to try on an older pc at some point but that will have to wait. I will follow these instructions then.
Thanks for all the replies and help with this.