First, the good news:
I successfully got the Dualshock 3 controller working through bluetooth. At first it didn't seem like it was working as it wasn't pairing. Most of the instructions I found on the internet says a box will come up requesting the passcode but this wasn't happening for me. Turns out the pairing isn't needed. Basically I just had to turn bluetooth on, press the "Playstation" button on the controller and my Macbook connected and recognized it as a PS3 controller. The LEDs on the controller continue to flash like it is trying to connect which can get annoying. To get the LEDs to stop flashing, I followed the steps mentioned in this post:
https://discussions.apple.com/message/20177064#20177064 I tested the controller with two native Mac programs: Cave Story and SNES9x. No problems once I configured the controls.
I tested the controller with Tyrian 2000 and Death Rally using Boxer. Again, worked fine.
Now the not-so-great news:
The controller does work in games which support DirectInput in Wine. My test game was Rogue Legacy. Turning DirectInput support on in the options allows the controller to work, but not in the way you will want it to. I wanted to map Jump to the X button and Attack to the Square button. Unfortunately the face buttons are not recognized. You would have to map these to the shoulder or other buttons. I confirmed the same with Shadow Man, the face buttons cannot be mapped (the analog stick does work btw).
This seems to be an issue not with Wine but with the way OSX handles the PS3 buttons & the Windows games themselves. The four face buttons are mapped to buttons 12-15. I'm guessing the games are not programmed to recognize such high-number buttons. I downloaded the free version of Xpadder and ran it with Wine and it correctly showed that buttons 12-15 were being pressed when I pressed the face buttons. So all button information is being properly passed into Wine, which is why I'm guessing the games are just not programmed to look for those buttons.
I'm thinking a program like Controller Mate might be able to get around this if it can be used to remap the buttons (say make button 12 register as button 4, etc). I downloaded the program but it is fairly complicated and I couldn't do a quick test with it. Will have to read the instructions and play around with it.
Rixasha: I don't know anything about Mac/Wine, but perhaps some of the same issues apply as do to Linux/Wine, so I'll state what I know after having spent a lot of effort to play windows games through wine using a wired X-Box 360 controller.
Thanks for the info! I tried running the wine joystick control panel but wasn't able to. I'm using a pre-built wine (1.5.23) via PlayOnMac so this might be why. Running "wine control joy.cpl" doesn't actually open anything and "wine control" just opens a control panel with "add/remove programs" and "internet settings". I've been thinking of building my own now that it has a Mac driver. The PlayOnMac builds still use the X11 driver.
EDIT: Forget about Controller Mate. It can't output controller button presses, only keystrokes and mouse. Of course I could use it to map the buttons to keys and just set Rogue Legacy to use the keyboard.