Yes, there are breakable walls in all environments and they look slightly different, e.g. breakable dirtwalls look a bit crumbly. Still, they're hard to recognize. In Dungeons it often helps to check the map for black areas, where you suspect a hidden room and simply hit the walls around it.

As DeMignon said, the walls look slightly different and can be (intentionally) hard to spot. The map usually reveals the edges of the secret area if you explore close to it, so even if you can't see breakable walls it's possible to know that area is accessible somehow. Sometimes it will be a moving wall rather than a breakable wall though.

Test this out by using the map near one of the secret areas near the beginning of the game to see it in practice.

Just look for walls that are different looking sometimes they look like stacked rocks.