First time I defeated him, I think he raised my "augment defense" skill. I died soon after and reloaded, and now each time I re-beat him, he raises my "augment damage", which I have no use for. It's vaguely frustrating.

I've heard that he randomly increases one of the other, but I've reloaded many times, or did some questing in-between, and he's stuck on "augment damage". Have I dreamed about "augment defense" ? Is there something that determines it (like the number of hits you take during the fight) ?

Tested and confirmed. Thanks.

If I get this right, you get the bonus point in ability with more assigned points? It sucks if this is the case as my Augment Defence is already at max and I was leaving A. Damage lower to improve it here.

When the game checks the levels of the skills, it should account for the possibility that your preferred (higher level) skill is already maxed. If not, you can always use iZakaroN's SaveEditor (or from Kiya's site, with description) to give yourself an extra skill point to learn Augment Damage.