I know who I am arguing with here but port knocking is not silly. It's fantastic.
When I say fantastic, I don't mean it solves all of our problems and obviates any other protections ... what I mean is, for almost zero cost[1] it adds a non-zero level of actual protection.
As a lifelong UNIX sysadmin, it is one of the few totally unalloyed security improvements that I have been able to add to my systems. I believe there are sshd vulns extant that you and I don't know about and port knocking allows me to worry less about them.
I also recommend SMS alerts on successful knocks - SMS alerts that you should never see in surprise. This is trivial, by the way, as you can put semicolons in the knock command:
/sbin/ipfw add 01021 allow tcp from %IP% to 10.0.0.10 22 setup ; /usr/local/sbin/timestamped_sms 4155551212 "knock from %IP% - "
[1] knockd on FreeBSD, 10+ years, not one hang or crash.