1) Images / files / etc. It all lives in cloud storage ("s3"), outside of K8s
2) RDBMS data. You can just run as hosted sql (say CloudSQL) or a not-in-k8s VM. I have found no compelling reason to move my RDBMS into my k8s cluster.
Category 1: stateless-"ish" workloads. More than 90% of hosts/containers used . . . less than 25% of operations headaches and time. Issues that happen here are solvable with narrow solutions: add caches, scale out, do very targeted, transparent fixes to poorly-performing application code.
Category 2: stateful workloads. Less than 10% of hosts/containers. 75% or more of operations headaches and time. Issues that happen here have less visibility, fewer short-term fixes ("just add an index and turn off the bad queries" only works so many times before you're out of low-hanging fruit), and require more expertise to solve in a way that doesn't require the application/clients to change.
If k8s and other next-gen technologies are only easing the first category, that makes me sad. It's like we have this sedan (off-the-shelf web technologies) that we have to take off-roading and it falls apart all the time. I don't want a better air conditioning system and more cushions in my seats; I want the vehicle to not break.
> Kubernetes is not aware of the deployment details of Postgres. A naive deployment could lead to complete data loss.
That sounds ominous, but is actually a tautology.
You have the exact same challenges anywhere else, but since K8s makes some operations so easy to do, you need to be careful. RDBMs are specially tricky because most of them expect a single "master" which holds special status. And it so happens to hold all your data too (as do your replicas, provided they are up to date).
Surprise! IT is hard and requires experts. Either hire one, or become one, but either way, the idea that "nobody in my enterprise/company/team need to understand the details of how stuff works" is crazy. Imagine a car-mechanic shop where nobody knows how an engine works: "we plugged in the computer tool, and it said something is wrong with your engine. I guess you need a new one"
Not everyone complaining about the deficiencies of current-gen tools is doing so out of ignorance or laziness about their stack.
I use k8s for bin packing and easy rolling deployments. Neither of which matter for my DB (i.e. no such thing as an easy rolling postgres deployment (maybe citris?)), and I am not going to put anything else on my database server... so no bins to pack.
If you are doing one or the other, then sure worry about it from a k8s sense. But don't throw stuff in k8s for no reason. It makes stuff like rolling upgrades of your cluster way more scary.
One of the cooler innovations we've seen, and I think we're going to see more of, is the ability to take a non-k8s VM and expose it to the cluster as-if it were just another pod. This would let you schedule and expose RDBMs and other specialized servers through kubernetes while keeping them on a standard VM.
I think that's the win/win approach to bridge the gap.
But I think that doesn't go far enough: I want to provide K8s with a YAML file and have Kubernetes itself go and create and provision a VM for me. I.e, I want to "kill" Terraform. I don't care about a 'fabric' network (although that could be convenient), just give me an IP that I can reach even it if is external to the cluster. VMs could be just one more resource that can be created or destroyed by k8s, just like network-based storage today.
I haven't found this exact use-case implemented yet. If noone else builds it in the coming months I'll probably start doing it.
The base pattern should be pretty easy to modify, although the use case here is very specific.
Also agree on keeping state outside. Maybe the relevant tech will be mature sometime soon but we’ve seen orchestration bugs do nasty things to stateless containers that would have been a nightmare if state had been involved.