Fire ON a wall would not do much, I agree. Fire UNDER a wall is a very different beast.

Digging a mineshaft under a wall, putting flamable stuff in the hole, then setting fire to it was the bane of medieval fortresses. It was not easy, and siege engineers had a dangerous job, but they were the quick-and-dirty way to destroy a castle, unless it was built on hard, rocky ground. (Which the Caer is not : It looks like it was built near a river, on an alluvial plain. The old dungeon itself, with its links to the maze, is rooted in the bedrock, but the younger external walls? Doubt it)

Gunpowder just scaled up the situation, rendering the classic medieval castles pretty much useless when it became widespread.

Maybe.
Still bandits...

Yep, bandits. Not combat engineers, but resourceful people used to dirty tactics, sneak attacks and hit-and-run tactics, that tend to get their hand over all kind of junk during their raids (including gunpowder). And probably a few old soldiers in the mix, too, if they go after a heavily defended target.


Frankly, there are FAR worse suspension-of-incredibility breakers in that game than this one. I'm a geologist working in construction and sewer engineering, and I must suppress an horrified shiver every time I "build" an extension of the castle, or every time I explore an underground dungeon. ^^