Cool thing about finalizers is that there's no straightforward way to delete them from the CLI, you need to explicitly patch the object to delete the finalizer so that goes a long way towards preventing instant mass deletion, they really do help a lot here.
The additional protection we have is via admission controllers which hook all API calls to prevent mistakes, of course, you have to foresee those mistakes and come up with a validation that prevents it. But even something as simple as "only 5 pods are allowed to be terminating at once" can help.
I absolutely agree about people claiming kubernetes is easy. It is easy to throw some helm chart at the kube API and suddenly have an app running with all it's databses configured. It is hard to keep them all running.
The pattern of operators seeks to solve this. Instead of a helm chart creating a MySQL statefulset, it creates a MySQLCluster that gets managed by a well-written operator following the best practices and correctly implementing all the hooks. The ecosystem is starting to converge on this finally and the beauty is that the sum of the industry's operational knowledge can be coded into these operators and we'll finally arrive at nearly fully self managing databases with backups and all.
The industry isn't there quite yet though, and so at least for stateful workloads you need significant operational experience. For background: I am the Tech Lead of our team that runs the kubernetes clusters and we provide all of the primitives our users need and control access. Each database type then has dedicated teams operating them and codifying operations into operators. There is a lot of work being done. But prior to kubernetes, all of these teams separately were coding against the EC2 API with no standardization, no unified view, ad-hoc failure detection, ad-hoc auditing, etc. Kubernetes is a very substantial improvement over that, but 90% of companies never reach a scale where this is necessary. But once extremely solid open source operators exist we may truly hit the dream of just applying some self managing operator manifest to an EKS or GKE cluster and getting an actually production grade database setup.
Version control is an interesting topic because it's hard to express transitive dependencies in that code. For example, the MySQL operator creating a StatefulSet and the Statefulset creating pods. It doesn't make sense to commit those lower level resources to git, they're not fully independent. However, for top level things, our build system produces artifacts from git and the deploy system creates them in the cluster. With this setup, only admins could directly apply them, which really helps with keeping things in sync with git.