ast486: You see the speed is for game purposes. You never wait in real time in a game or it would be horribly boring.
That's my point, actually. ;)
Both "building these walls is terribly easy" (in price, manpower and time) and "destroying them is terribly easy" (in number of attackers and their equipment) ARE for game purpose. If you wanted the game to be completely realistic about it, there's no way you could finish even one building before the game came to an end.
So either you accept it for gaming purpose and try not to think too hard about it, just like you did for construction (and which I tend to do too, because realism is often boring), or you justify it in your head like I just did ("there are really a fair number of bandits with good equipment, but we only see a small part of the battle", "the construction was old and crappy", "they managed to sabotage/infiltrate and detonate the castle's gunpowder reserve", etc...)
Although I agree that one full fledged siege battle (like the one in NWN2) where you can feel your castle's power in a less abstracted way would be nice :)