Do you have any learning recommendations for someone looking to start down this path? I've only ever worked in an infra-renter context, and I've begun exploring the 'rent from Hetzner, manage your own infra' for personal projects, but I would love to learn from the paths of experts where possible.
Ideally, you also have some exposure to this at $job as simply building DIY infra horrors without seeing the real-world context, tradeoffs, etc. in which they typically operate will be misleading.
1. a DNS monitoring with failover (DNSMadeeasy has a decent solution)
2. a Haproxy setup with health checks for switching to a working upstream service
3. a distributed filesystem
4. a master-slave replication with monitoring (something like Mariadb + orchestrator service)
and nightly backups for all this. Database and FS are latency sensitive so they shouldn't be too far apart.