Everyone else is chiming in with their anecdotal proofs, so I will with mine.

Yes, if you have a long password or key-only method of logging in, and disable root login, and patch the daemon, then party on.

But for the extra bit of protection against 0-days and bullshit amounts of logs from quick port-22 only scans on your network, it helps to move it to another port. This is why I disagree with your mathematically founded reasoning. It doesn't fit with real world scanner behavior.

If it's just your home network, it's not confusing because only you use it. If it's a corporate network, then using putty might be a little annoying, but it's not hard to communicate another standard.