You are not a good player.. i am da best of the best..... Sultan Abdul Khalid XXIVVIX

Well a rather old post brought back, but a valid point. However all games use unrealistic elements (ever stopped to think that most buildings are far out of proportion to the size of your soldiers in most RTS games) the trick is to understand what is happening.
Whilst your opponent can instantly rebuild walls and towers, doing so costs resources each time. So hit his resource points outside his walls - lob flaming arrows and cows over the walls to cripple any production going on within and drive down his ability to make money. Try to target and take out any markets as well - if you run your opponent out of stone they can still sell and buy resources at the market - run him out of wood and destroy his market and he's stuck with no way to get more.
Eventually hitting your opponent enough for long enough will run them out of resoureces (this assumes you keep your stone resources coming in to fuel your catapults). In real life this could equate to the long drawn out length of a siege - with many taking days or weeks to complete.
If you want a (very slightly) more realistic approach, but minus the castle building look to the Total War series of games. Otherwise as far as I have found Stronghold is the only fantasy RTS with a decent castle building side to it. Many others might offer some walls and towers but that it about it

Also of note...
If you have soldiers situated nearby the location of the tower being destroyed, like attacking the walls next to it or thereabouts, the enemy should be unable to rebuild the tower.
This also applies vice versa to your tower/wall rebuilding.

Yep. It might be a tad unrealistic but what game is? I'd actually hate to see an realistic and accurate game!
And do what both members before me suggested, that should work.

Ahh yes I forgot about the troop ranges - yes as Darkness said getting your troops close will prevent them from rebuilding walls and buildings near to those soldiers. This can mean that you have to rush the walls just before they go down so that your men are in position to storm through as soon as they are breached - rather than breach at distance and then approach. This of course puts youre men at more chance of being cut to bits by defences - so remember you have other options like siege towers, arrow guards and with the expansion the power of ninja to storm walls and take out archers.