As others have said, I don't find these arguments very convincing - if castles were mainly residences, then why have crenelations, or arrow slits or a moat, doesn't that argument apply equally to those? The argument that if such a defensive feature was needed, then its already too late, and so it was probably never used is also not a good one imo. Has the CWIS gun on aircraft carriers ever actually been used in combat? Am I to conclude that CWIS has no defensive use and is purely architectural ornamentation?

The only good point is that it wasn't mentioned by contemporary sources, but I don't think this proves anything unless you can show that other defensive features on castles were mentioned. Otherwise maybe we just have bad sources about castle design. But if you can show me some documents about why eg. moats are designed they way they are, then I think you have a point.