We'd need something more like a cryptographically attested setreuid() and execve() combination that would run only commands signed with the private key of the intended user. You'd want to use a shared clock or something to protect against replay attacks
Whether it's worth it or not depends on circumstances. In many cloud environments, nuking the VM instance and starting over is probably easier than fiddling with SELinux.
at minimum, ipv6 only if you absolutely must do it (it absolutely cuts the scans way down)
better is to only host it on vpn
even better is to only activate it with a portknocker, over vpn
even better-better is to set up a private ipv6 peer-to-peer cloud and socat/relay to the private ipv6 network (yggdrasil comes to mind, but there's other solutions to darknet)
your sshd you need for server maintenance/scp/git/rsync should never be hosted on ipv4 clearnet where a chinese bot will find it 3 secs after the route is established after boot.
It's not like this is SSH's fault anyway, a supply chain attack could just as well backdoor some Fortinet appliance.
I'd stop at SSH, since I don't see a conceptual difference to how a VPN handles security (unless you also need to internally expose other ports).
The only software remotely in the same league is a stripped down Wireguard. There is a reason the attacker decided to attack liblzma instead of OpenSSH.
If you have password authentication disabled then it shouldn't matter how many thousands of times a day people are scanning and probing sshd. Port knockers, fail2ban, and things of that nature are just security by obscurity that don't materially increase your security posture. If sshd is written correctly and securely it doesn't matter if people are trying to probe your system, if it's not written correctly and securely you're SOL no matter what.
I fail to see a problem here.