Yep! And I should be clear: I am not saying just don't change the SSH port. I'm saying if you care about security, at a minimum disallow public access to SSH and set up a VPN at a minimum.
>Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them).
I'm a bit confused here. In every major distro I've worked on (RHEL/Cent, Ubuntu, Debian, SUSE) the default httpd and nginx packages are all configured to use their own user for the running service. I haven't seen a system where httpd or nginx are running as root in over a decade.
I think the bare minimum for anyone that is running a business or keeping customer/end user data should be the following:
1) Only allow public access to the public facing services. All other ports should be firewalled off or not listening at all on the public interface
2) Public facing services should not be running as root (I'm terrified that you've not seen this to be the case in the majority of places!)
3) Access to the secure side should only be available via VPN.
4) SSH is only available via key access and not password.
5) 2FA is required
I think the following are also good practices to follow and are not inherently high complexity with the tooling we have available today:
1) SSH access from the VPN is only allowed to jumpboxes
2) These jumpboxes are recycled on a frequent basis from a known good image
3) There is auditing in place for all SSH access to these jumpboxes
4) SSH between production hosts (e.g. webserver 1 to appserver 1 or webserver 2) is disabled and will result in an alarm
With the first set, you take care of the overwhelming majority of both swarms and persistent threats. The second set will take care of basically everyone except an APT. The first set you can roll out in an afternoon.
With the first set, you take care of the overwhelming majority of situations.