Paradoks: You only destroy enemy buildings by conquering the land where they are placed. In other word you must capture (not destroy) surrounding military structures (or, if all are destroyed - building and occupying your own military building near that area). I'm not sure but I also think (that's definitely how it worked in Settlers 3) that destroying all of player's military buildings eliminates that player from the game.