One of the flaws of Crusader is that it's not so newbie-friendly. The original Stronghold is much more in-depth as it gradually introduces new elements (yet most of that only happens in the combat campaign). If you don't know your way around it can be a bit tough at first.

While the economic-based side of the game forgoes most combat, the occasional roaming raiders/wild animal pack does still happen as you've found out the hard way. Walls will still be your friend for your castle center. If possible it's always good to have towers manned with archers/crossbow men nearby sensitive installations. If that's not a real option, any more elevated ground ought to do. Foot patrols by pikemen/halberdmen may also be needed.

Also, you want to take note of any signposts on a given map since those are the points where raiders/packs will pop up when they do. Their approach can be a little unpredictable at times, but at least you'll have a general knowledge of what their attack vector may be, so you can prepare for that.