One Hander gives you an accuracy bonus when using one handed weapons, such as pistols, unarmed weapons, and certain melee weapons. But it also reduces accuracy when using two handed weapons. But is it even worth taking? Or should I not bother?

Assuming we're talking F1 and F2.

It will depend on the character you're playing.
You're biggest downside is maximum range. With small guns the max. range on a rifle is 50 and on a pistol it's 30. With energy weapons that's 45 vs 35.
The exception is the gauss pistol in F2 which has a range of 50. Downside is that ammo might be hard to come by.

All other downsides can be fixed with special, thraits, skills, perks and armor.
You could go for a high luck build combined with finesse, or a high luck build combined with fast shot.
Both combine great with bonus rate of fire, more and better criticals and action boy.

A waste of special and skills would be to compensate the penalty on 2 handed weapons by raising intelligence and putting extra points in your weapon skill of choice.

So, yes or no?

It's fun to play and worth taking if you decide to go for it.

10 agility, fast shot, bonus rate of fire and 2x action boy gives you 6 shots with a gauss pistol and 14mm pistol.
Combined with high luck (6 or 10) and more and better criticals, you can do a lot of damage.
In F2 you can take jet for another +2 action points, so you can reload the 14mm in the same turn.
Since you need to get closer to your enemy some extra endurance and damage resistance might be worthwhile. You can use buffout and psycho for that till you get a power armor and after you got 1.
An upside compared to 2 handed weapons that most pistols take 4 or less strength, 1 less than most rifles.

I once did a playthrough with a high luck small guns pistoleer, though I think I took finesse instead of fast shot. It was fun to play with. All that was missing was a cowboy outfit.

For your typical F2 build of small guns;lockpick;speech One Hander is not very useful. You're just sticking to small guns, you should have enough skill points to attain good accuracy on a single weapon skill.

That said it's useful if you're playing with more weapon skills, there it can save a lot of skill points i.e. in F2 you can try a build where you start with throwing and use flares/grenades+one hander, then on lvl 13 transition into energy weapons with alien blaster+one hander and some unarmed for the boxing quests. That's obviously a very meta build, but here One Hander saves you skill points across 3 weapon skills (throwing+energy+unarmed) and also saves you additional skill points that you would lose on the F2 skill inflation thing. Technically you could level up your small guns skill with books and there you'd have long range fighting capabiity with the gauss pistol + One Hander (without investing a single skill point into small guns).

So that's where I'd try to fit in One Hander - useful for builds that try to use one handed weapons from different skill groups. If all you're trying to do is to use revolver magnum and then switch it to the gauss pistol, One Hander's contribution is negligible.

So in other words, it's not a useful trait and I should just skip it?

If you're playing the typical gifted sniper I'd not take it.