What determines whether a unit can counter an attack? In other words, return an attack without being player controlled?

Unless the attacker has the "no enemy retaliation" ability (like Sprites or Vampires), then the defender will automatically counterattack against the first melee attack.
The exceptions are Gryffins (retaliate 2 times), Royal Gryffins (retaliate against any number of attacks) and units under the effect of the Counterstrike spell.

Which HOMAM are we talking about here?

The default is that each unit can retaliate against 1 melee attack per round. The exceptions to this are what klaymen covered- units that can't be retaliated against, the few units that get multiple retaliations per round, and the Counterstrike spell (HOMM3 only, if memory serves).

sorry - HOMM 3

Upgraded harpies (harpy hags) are another one that can't be retaliated against. Plus they return to their starting spot after attack. Tremendous units because of these abilities. They can attack and be safe from multiple units one after the other this way. Get slow on an enemy and harpies are brutal.
Also, there is a spell -- I think it is counterattack, of the air school, that allows for double retaliation.