When I do combat as a brigand the sword icon (where the bladerang icon is if you're a rogue and where the spells are if you're a sorcerer) doesn't seem to do anything. How do I get that to work?

Block. Set blocking, after 3 succesful blocks you will get special hit.