So, how do you win this game? When does it end?

Eviction notice to all competing races in the sector (destroy every alien colony), after you kill all colonies there will be a message that alien race has left the sector and their icon will no longer appear under the sector map.

Maybe you also need to destroy all their transport ships that start colonies (they can be a part of fleet), have not tested this.

Note they will flee from you and start new colonies as long as they have money and transports to do that, chase em.

it's not actually enough to kill their colonies. You also have to have killed all their transports. I had a case where I wiped everyone out and as I destroyed the VERY last Mikotaj colony the game did not end.

They had a small fleet escaping and in that fleet were two transports. The fleet ran out of fuel, dissolved and the two transports were lost in space. Only then did I get the "Mikotaj forces have withdrawn" and the game ended.