Currently I can lead around 80+ soldiers in my unit, and they are riding/marching with me.
However, I own one castle, and it's unprotected, so I need to know this: do I lose the number of troops running with me if I garrison some of them within the castle?

In other words :-) : does that number of 80+ people that I can lead mean I have to distribute them wisely, or can I have 80+ garrisoned as well as 80+ riding with me?

AFAIR, you can garrison as many troops as you like (and have the money to pay). They will remain in the garrison indefinitely, draw half of the standard salary every week and die only, if your castle is attacked.

The limit is for your riding party alone.

Confirming what -Iota- said. Let me add that garrisoned troops do not eat.