Yes, all wands drain the user.
The advantage is that they allow you to cast high-level spell with 100% success rate, regardless of user's magic skills. IMHO, it's good to keep one or two as a 'last-resort' item for some tougher fights, at least at earlier stages of the game.
On the other hand, it's pretty easy to get some higher grade equipment and level your party early, if you have enough patience for that. Or you can always reload the save when the fight goes badly, so If you don't like the idea of losing some skill points, just sell them.