I recently upgraded to an X56 Rhino Joystick HOTAS and I'm having a heck of a time getting Privateer and WC3 to recognize it. Is there a trick to this?

1- Privateer and WC3 don't have the throttle, it's either keyboard or bust, even with programmable controllers the lack of speed presets makes making the throttle work a pain (no such problem with X-wing as it has speed presets so it's easy to program the throttle even if it's not supported).
2- that said the stick should be recognized by dosbox as long as it has the priority in the windows control panel under gaming peripherals (where the menu is exactly varies with the windows version, but it should be somewhere in Control Panel).
3- the Rhino sometimes has driver issues, it's probably not your case but usually less fancy peripherals (or fancy peripherals from more competent brands like CH Products or Thrustmaster) have less problems.

The Win32 executable of WC3, which is compatible with the GOG package, does have throttle and overall HOTAS support

The Kilrathi saga version?
Since I never managed to get the first one from that compilation to work on any Windows OS after 98 (I had it from a PCgamer CD) I assumed it would probably be the same for WC3.
I'll keep it in my mind for my next playthrough.
It would be nice to have that version as an option from GOG though.

Yes... there is a package linked somewhere in another post here, that upgrades just the EXE for WC3; worked pretty good on Win7 x64 an I think I tested it also on Win10

Oh, goody, I'm downloading the package right now lest it gets lost by the time I get around to replay the game.
Now if only someone did something similar for the first two Wing Commander games...