You don't have to leave anyone there, I think I only did it once and found it really didn't make much of a difference. I noticed on a few occasions that random friendly troops just seem to station themselves anyway in my castles without any input from me. Sometimes the castle will be attacked but it will be easy to retake as there will only be a handful of enemy troops there (if any). Save your battle hardened troops to take with you for the main fights :)

Thanks - I'm glad. I really like the game, but I am not looking for an RTS strategy type experience where I have to spaz out that I keep all my properties protected at all costs. Combat is great!

If you leave troops in your castles or towers, they will gain experience each day/week (not sure which) and train up over time. So I always stuff them full so I have a pool of experienced replacements to draw from if needed. It is sometimes way cheaper to replace a unit with many wounds than to pay to heal it if you have a similar level unit in a tower to draw instead.

They also will defend your castle/tower so you don't lose your accumulated income if you haven't gone there to collect it for a while. Only takes a few minutes to stuff them with troops and saves a lot of back and forth moves to retrace your steps and retake it from roving bandits.