Or possibly set up port knocking, may be overkill though
I compromised on some of my boxes and made a non-published/non-linked URL that if you hit it, your source IP will end up on the SSH-allowed list. That plus a non-standard SSH port seems to keep 99.9999% of the risk out of the way without being too overly cumbersome to knock-through when needed.
[EDIT] You could always just set up iptables to accept traffic from your ISPs subnet and deny the rest.
Only one knock is needed to keep the port open for my current IP address indefinitely, or it can be closed immediately without losing my ssh connection.
It also precludes the necessity of filling up iptables with boatloads of DROPs to keep out the riff-raff.