I have a Logitech extreme 3d pro controller. I'm able to use it for flying and shooting, but none of the other buttons it has does anything. I like to map several of the programmable buttons the joystick has, but the menu does not list any controller/ joystick settings. I looked at forums where I saw people say use Cltr+F1 or go to the joystick settings in game, however, neither the button press does anything nor is there a joysticks settings I can find.

How do I map joystick controls / change settings after the prompt for joystick calibration? I'm on Mac OS Mojave 10.14 and playing the game through GOG galaxy launcher.

Short answer: you can't.

Long answer: dos games rarely supported more than four buttons out of the box due to hardware and software limits and didn't allow customization, joysticks with more buttons either were from bigshots like thrustmaster that sponsored some games to have support out of the box or came bundled with a software to map the extra buttons, often using a PS/2 passthrough to allow the extra buttons. Now the logitech joystick does have a software like that but IIRC it only works under windows.

The windows version allows to map joystick buttons (at least Tie Fighter does and the joystick menu isn't very intuitive to find) but it's also a rather drab expericen with a lot of transitions stripped away and no dynamic soundtrack, just a couple of CD audio tracks looping.

There are however several softwares that can map joystick buttons, be aware however that often cannot remap already assigned buttons in DOS because they are hardcoded (the bundled softwares get around it acting at driver level) and I'm not sure I know any that support macOS.

For which of the 3 games exactly are you asking for?

I managed to fix it with that exact joystick.

find the game folder (it was c:\Program files (x86)\GOG Galaxy\Games\Star Wars... on my machine)

make a copy of the file dosbox_tiecd.conf before modifying it. (In windows, ctrl+C then ctrl+V is sufficient).
Edit the original.
in the joystick section, this is what I have.

joysticktype=2axis
buttonwrap=true
#timed=false
#autofire=false
#swap34=true
#buttonwrap=false

who knows why those other options were set. This worked perfectly for me.
I also used Logitech's mapping software to set the buttons on the joystick, e.g., weapon select, target select, all those invaluable things.

Just in case it's useful, I also made the changes so that it actually does aspect correction. Dosbox config is super confusing.

fullscreen=true
fulldouble=true
fullresolution=desktop
windowresolution=original
output=openglnb
pixelperfect=true

It also looks like pixels, not rounded annoying things.

Finally, you can google around to improve the sound synth if you like