It was painful. We quickly realized that rolling upgrades often got blocked. The reason being peristent volume claims sometimes get stuck.
And this is just one of few problems.
Since we moved databases and stateful services outside Kubernetes, everything got faster and more reliable.
I personally haven't experienced the issues the OP is describing. It might be that StatefulSets and PVCs are more stable now then when OP tried them. Of course using managed/hosted K8s makes a big difference.
There might have been a way to support our use case better. But this was managed service and we couldn't easily tweak every settings. But also, we didn't want to.
Our goal of using Kubernetes is to make difficult stuff easy. Not to make easy stuff difficult. And in our skillset and capacity moving stateful services outside K8s was easier, cheaper and more reliable.
K8s is great way to easily deploy and scale services. As we leverage Kafka, a lot of applications fallback on Topics and KTables for storage.
Kubernetes with jobs, cron jobs, operators, secrets etc is effectively an Operating System for the cloud.
Definitely worth the investment to learn and practice.
The fix was to add scheduler hints that moved the pods to the correct zone as I should have done in the beginning. (On first deployment, the disk is created on the correct node and thus zone, so it all seemed to work).
Just curious - how does GKE control plane upgrade break DNS?
The cluster's DNS server refuses connections briefly during control plane upgrades or updates. I haven't tried their node local DNS yet (enabling it will cause downtime, natch), maybe it'd help.
Either way, this is a good example (for me) of how Kubernetes needs some work to be friendlier to operators