I really want to like Stronghold HD, and mainly build castles with a bit of protecting it from enemies on the side, but these ground units like bandits or whatever do so much damage.

Is this just not the game for me? Is there a way to nerf their damage? Is there any other games that you might recommend? I mainly want to build awesome towns and castles and such. If only the enemies couldn't tear down walls so easily..

I've been playing this game for over 10 years and there are several things you could do.
Try pitch in small blobs (not connected) a long your walls which you can set on fire with fire shooting archers. These 'bandits' will die very quick. Boiling oil pots will do too (engineers), throwing them down from the walls).

The Bandits are probably Macemen. They're quite fast and do lots of damages, but their armour isn't the best (it's good though).

You could try Castle Builder mode if you haven't already. It's lovely for the type of game you want to play (and with F1 and F2 you can start events).