Does it use DOSBox, like the Windows version does? If so, there are some DOSBox settings you can change to try and get the joystick working. Check out the following video, it should still apply even though it's running under Windows, since it's mostly about DOSBox settings.

The video is linked at the top of this post:

http://www.gog.com/forum/star_wars_xwing_and_tie_fighter_series/xwing_1994_tweak_guide_video_graphics_sound_music_speed_and_controls/post1

Well according to my account, the only versions of X-Wing that have Mac support are the 1993 and 1994 versions, which both ran under DOS. On Windows, both of these run through DOSBox; I assume the same is true on Mac.

Can you check the install location and see if there is a file with the .conf extension? If so, that's the DOSBox configuration file for the game, and you can make certain tweaks to the joystick settings in there to see if it gets it working. These tweaks are covered in the video I linked earlier.

Then again, maybe GOG managed to get the original Mac release and get it running on OSX? Anyone know for sure?

It would be Boxer standalone on Mac.

No. The original is for PowerPC Macs, Mac Classic OS, and requires a ROM to run on Sheepsaver or the like.

I think HiPhish posted something about Macs & Joystick problems in some thread in General recently. Don't remember which one though.

I see. I'm unfamiliar with Boxer so I'm not sure what to suggest. Hopefully someone else will have some ideas.

There you go:
http://www.gog.com/forum/star_wars_xwing_and_tie_fighter_series/xwing_games_joystick_and_keyboard_setup