About the final score, it seems to require high fame. However, when my army is on the march I go around razing as many cities as I can. If I banish him I don't want him to have anything to come back to, plus I just want to keep moving and not stop to build up and defend everywhere. But that kills my fame and my final score, doesn't it?

Yes, and it also reduces your final score because you don't have all of that population to count towards it.

OTOH, if it allows you to finish the game more quickly then the time-based bonus will be higher, and this may counteract the loss.

Almost everything contributes to final score. Number of spells learned, turn finished, difficulty level, number of citizens, number of enemy wizards banished and whether you cast the Spell of Mastery, total fame.

Try to hang on to enemy settlements. Conquering a city intact gives extra fame and those citizens are breeding VP. Also the AI will try to take them back and those victories will earn fame as well.