Short version:
Client and server share a key.
Each can generate a TOTP based hash.
Client sends hash to server listening on UDP.
If hashes match the server opens the normally closed ssh port for a few seconds (long enough to make a connection).
Like a Google Authenticator TOTP code, the correct hash changes every T seconds so identification of the UDP port and interception of the key is only helpful for a limited (if any) amount of time.
Is this worth turning into a robust daemon? Is there a better way to deal with constant ssh probing? A module in a firewall would be ideal. Environment based config would make it fairly easy to use in provisioning for ssh admin with a smaller scan footprint.