There were interior defenses though. In some of the more impressive castles there were concentric rings of walls so breeching on wouldn't surrender the whole castle. There were often also keeps. Even in early proto castles you had motte and bailey designs where the main walls defended the low ground (bailey) for an interior fort built on a hill or prominence (the motte).

Castles weren't single layer defenses that stopped at the walls, they were definitely designed to continue the fight and protect the nobles even if the enemy did get through the walls. Given that it's not so out of line to think the interior architecture might have been designed to increase defender's advantages.