Your argument is the main point he makes in the intro and is the strongest argument against the theory. But the arguments the author lists himself are not very strong:

1. "The swordsman theory is predicated upon the belief that castles were intrinsically designed as military structures" => "just because most were; there are a lot of castle like fortress that were actively used as defense sitations.

2. If staircases were genuinely engineered for military purposes, then we should always find that they turn clockwise. => No, opposite, since not all were used military it might not have been a priority for all, and also not all builders might have been aware of this strategy.

etc...