https://www.youtube.com/watch?v=MqF63EsJRlQ video of the process from a fresh install

Fullscreen mode:
download dgVoodoo
extract somewhere
launch dgVoodoo and press the Add button on the right of config folder and point it to the R6 folder
General tab: appearance fullscreen,scaling mode unspecified,capture mouse. pretty sure its just default for this screen
DirectX tab: videocard dgVoodoo 3D, vram 1024, if you want turn filtering to max and msaa to 8x.
Make sure you set resolution to unforced otherwise the game will blur when you move the mouse around in game
press Apply
go to dgVoodoo folder and open MS folder and copy the 3 files D3D8,D3Dlmm,DDraw.dll and put them in the folder where rainbowsix.exe is

go to the registry and find r6 mine was located under
HKEY_USERS Software\Classes\VirtualStore\Machine\SOFTWARE\Wow6432Node\Red Storm Entertainment\Tom Clancy's Rainbow Six

Fullscreen TRUE
VideoResolution 1920x1080

while you're here change MouseRadiansPerPixel to 0.002727 or whatever setting is fine for you by default i think its 0.008727 which makes your crosshair really unstable and the sensitivity through the roof

I would also suggest capping the games framerate to 60 by whatever means. I use nvidia Inspector for mine r6 sieges profile works for the original game for some reason. if you don't the AI both enemy and friendly will often run in circles
2-Sync and refresh
frame rate limiter 61.8

Windowed mode:
follow the above up to the registry step but change
Fullscreen to FALSE
VideoResolution 1600x900
do not use your max resolution for this if it goes out of bounds it will crash, you can make incremental changes if you want to make it as close as possible.

the menu part of the game will still be small but once you load into a map it will change to the desired resolution

the only compatibility setting enabled is 16 bit colour

I really hope this works for you, if not tell me what the error says in the Sherman.log file located in the folder with rainbowsix.exe

Tried it, a "Could not launch due to an internal error; try rebooting your machine or reinstalling Rainbow Six." error. Tried it both on the gog and ubi versions. Log file says:

Renderer::Initialize() failed during DirectDrawEnumerate() in Renderer::CreateDirectDraw() in file C:\Develop\Common\Renderer\renderer.cpp at line 527
Error = "Generic failure."
We are sorry, but we are unable to initialize the renderer at 640x480x16
Renderer::Initialize() failed during DirectDrawEnumerate() in Renderer::CreateDirectDraw() in file C:\Develop\Common\Renderer\renderer.cpp at line 527
Error = "Generic failure."
Renderer::Initialize() failed during DirectDrawCreate() in Renderer::CreateDirectDraw() in file C:\Develop\Common\Renderer\renderer.cpp at line 539
Error = "Generic failure."
Renderer::Initialize() failed during DirectDrawCreate() in Renderer::CreateDirectDraw() in file C:\Develop\Common\Renderer\renderer.cpp at line 539
Error = "Generic failure."
We are sorry, but we are unable to initialize the renderer at 640x480x16 either
GAME: Could not create Display object
Displaying fatal message box with the following message:
Could not launch due to an internal error; try rebooting your machine or reinstalling Rainbow Six.

just to clarify you are on w10 right? win7 and below only need to do the registry steps from memory

After that we need to check if you have ddraw stuff installed by doing the following
Open Run Command (you can do it by pressing WinKey+R) - > type: control - > press ok - > go to Programs - > Programs and Features, click the Turn Windows features on or off link, find the Legacy components on the list, enable DirectPlay and click OK to install it.
restart your pc and then try the game again

and lastly to double check if dgvoogo config is in r6 folder, dgVoodoo.conf is file name

I am on win10, yes. I do have ddraw, and the config file is there, too

after a clean install it seems to be working well, though one bothersome thing is that in all the menus (including planning), the mouse and other 2D elements that move around leave a trail behind. What could be causing this?

Thank you for your help.


EDIT: I didn't notice, the change in resolution didn't take, regardless of whether I use the registry value change or if i force the change through dgvoodoo, it seems to still run at 800x600

regarding the blurring when moving the mouse around is caused by forcing msaa or aa in dgvoodo so disable that sorry but keep the resolution setting unforced
registry setting and other stuff

I hope this works since I created this myself,tested it and it worked on my pc. but unzip that somewhere and open either the fullscreen or windowed and allow it to change your registry, I've also added the dgvoodo files just in case you want to overwrite yours.

just for safe measure set the compatibility for R6 to Windows 98,16bit (this one is a must) and run as admin

tried it, still stuck at 640x480. Also, don't know if it's related (most likely not but you never know with these games), but during the first mission, one of the hostages completely fails to spawn in.

I'm glad this is such a good game otherwise I'd be sick of jumping all these hoops to get it running, haha

hmm can't say i've experienced the missing hostage on mission 1, as you would already know there is no secondary hostage on easy but follow this to fix it
https://pcgamingwiki.com/wiki/Tom_Clancy%27s_Rainbow_Six#Invisible_NPCs

If you could send me your registry for R6 I'll have a look through it just in case
My last idea as to why it's not working might be because there's another registry entry that the game is reading from. If you could search for Rainbow Six and mash F3 until you find it

pcgamingwiki listed this as one of the locations HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Red Storm Entertainment\Tom Clancy's Rainbow Six\

disable software mode in the gfx menu, that should stop the mouse trail problem.
in dgvoodo point it to r6 folder again, general tab - scaling mode - stretched, keep aspect ration
My very last suggestion would be to download Rainbow Six Black Ops which combines all the early R6 games into one and uses the Rogue spear engine i think and you can use the fix in the RS thread for it

I imagine it requires Rogue Spear, right?

Nope it's a stand alone game

oh, nice. I'll give it a go.

dgVoodoo almost gave my computer a virus, so yeah, no.

How does it almost give you a virus, what a load of garbage
edit, upon further looking theres a bunch of false positives that the creator even put a message about it on his site and theres plenty of other threads on gog talking about it

I'm naturally cautious about these kinds of programs, and I trust my anti-virus software. Can you assure me that the program is safe?