I've been running my home server with an completely open SSH port since 2012 and haven't been hacked yet. And I know since I check occasionally with Tripwire and/or RPM verification from a clean boot.
The only downside to an open SSH port is the thousands and thousands of log spams from connection attempts. The upside is that I can log in from anywhere using my phone directly or as a hotspot for the laptop.
Oh yeah, and VPN just pushes the problem back one level. For me SSH is the VPN. Requiring VPN access first to get to SSH is just pushing any security problems back into the VPN server. Because now IT is the one with the open port to the internet.