Before doing something like this I would worry a lot more about client endpoint security (exactly to what level do you fully trust all the people and workstations/laptops that are authorized to ssh to this thing?), as an overall more likely threat.
There are also lots of less esoteric ways to not have a system listen on any publicly accessible IP address whatsoever. If it's really something critical you should be looking at a combination of making it purely an intranet service only, listening on an IP in an internal network block that isn't accessible from global routing tables at all. Or is completely firewalled off from the world, and only accessible once you've authenticated yourself to your VPN. Or only reachable once you first authenticate (public/private keys, two factor crypto key auth, etc) to a bastion host, and then reach the system from the bastion.