Is port knocking a common defense these days? I remember reading about it awhile ago, and thought it was a neat idea but didn't really think people would actually use it.
I compromised on some of my boxes and made a non-published/non-linked URL that if you hit it, your source IP will end up on the SSH-allowed list. That plus a non-standard SSH port seems to keep 99.9999% of the risk out of the way without being too overly cumbersome to knock-through when needed.
[EDIT] You could always just set up iptables to accept traffic from your ISPs subnet and deny the rest.
Only one knock is needed to keep the port open for my current IP address indefinitely, or it can be closed immediately without losing my ssh connection.
It also precludes the necessity of filling up iptables with boatloads of DROPs to keep out the riff-raff.