I'm using a PS3 controller, and am finding it hard to figure out the best control set up, any advice?

A joystick...

Need a joystick.

P.S. From what others are saying, it's really going to want a joystick. I originally played the 94 version with keyboard and mouse and it was good enough, though not great. But you'll really want a joystick to enjoy the game. A controller doesn't work very well because of how sensitive and awkward the joystick on the controller is. But if it's the route you want to go, then I'd recommend setting a button (probably triangle) for target (T key), some bumper buttons for next target and previous target ([, ]) and square or circle for change fire type (can't remember that key), X button for fire and square or circle for changing fire type (single-shot, dual-shot, 4-shot).

I would recommend keeping the engine, shield and laser power controls on the keyboard (I think those are F9-11, if I'm not mistaken).

I hope that helps somewhat.

Nonetheless, I'd still recommend getting a joystick. If you want another reason, after you'd done with TIE Fighter, you'll need it for X-Wing vs TIE Fighter (surely getting released, I'd imagine) and lovely games like IL-2.

Yeah, a joystick with square axis limits is preferable especially for the DOS versions.
If you have round axis limits I suggest to go to windowed mode (Alt-Enter), open the gamecontroller panel and use the calibration panel so you can see if you're getting the corners right during the calibration routine of the game, after that you can return to fullscreen using Alt-Enter again.

The sticks on the playstation controller is got to be the worst kind for this style of game. You'd probably get better control from your mouse and keys.

Use what I used in the 90's, a Gravis gamepad! 8-way D-Pad and 4 buttons. You had to have 4 arms because you also needed to use the keyboard.