Hacker News Reader

Comment by markbnj | Hacker News Reader

> How do you deal with stateful applications like databases?

The impedance issues between kubernetes and stateful applications have been mostly solved. The main problems stemmed from the object model not yet being mature enough. Deployments did not offer good solutions for assigning stable identities to pods across restarts, or making it possible for them to find and attach to persistent volumes. StatefulSets and PersistentVolumeClaims solve these problems. Most of our k8s workloads are stateless, but we do run some big elasticsearch workloads in it. Having said that, I think if you're a small- to mid-sized business running on the cloud managed offerings are the first place to look. GCP CloudSQL has worked extremely well for us.

View on HN

To someone who is not familiar with the technology beyond a high-level overview like in OP, the sentence "The main problems stemmed from the object model not yet being mature enough" sounds like "The main problems with stateful applications stemmed from there being problems with stateful applications".

The problem was that these features came a little later after many of the stateless features were in there. Now we're talking about the stateful stuff being stable for many years, so all the arguments against it are usually old or uninformed.

Fair enough. I could have said it more clearly. The strategies, api entities, and implementations to support stable pod identity and declarative persistent storage weren't there a couple of years ago, but now they are.

What about a on prem kubernetes cluster, would you use kubernetes for something like a MySQL database, or run it outside or the container environment and just link to it from the stateless services in the cluster?

Not OP, but my team is running on 100% k8s, mostly on bare metal. We do everything on k8s because of principle of least surprise: When apps are running as Deployments or StatefulSets, everyone knows how to deal with them (how to find logs, how to restart etc.) and it fits in the established deployment procedures (everything is a Helm chart etc.). When you have a single stateful service with a completely different configuration (e.g. RDBMS on a dedicated machine), it's always going to be "the odd-ball that only Greg knows how to upgrade". The whole point of k8s is not having these odd-balls. We have all databases in k8s, with physical volumes backed by an NFS-connected storage that's sitting in the same rack.

> We have all databases in k8s, with physical volumes backed by an NFS-connected storage that's sitting in the same rack.

I'm surprised you haven't seen any major issues with doing this. NFS exercises some dark corners of Linux's VFS, and most people don't run databases on top of it (meaning you could run into a fair few pretty fun issues once you're doing enough transactions). From memory, most database vendors are pretty twitchy when you ask "do you support running $DB on NFS?".

> We have all databases in k8s, with physical volumes backed by an NFS-connected storage that's sitting in the same rack.

I was searching about this yesterday, how is your performance running a database in NFS? If your entire db can fit into the RAM cache it's probably fine, but when you have to do heavy IO is what I worry about with NFS.

I think it depends on your NFS. A storage appliance with NFS hardware offload will work much better than a software NFS.

Personally with bare metal database I'd try to do fibre channel if supported, then iscsi. Would only dabble with NFS if it had hardware offload, but even still I think block storage would have better iops.

AFAIK the reason for using NFS is that it's supported in k8s out of the box. I don't recall any performance issues for the things that we're doing. When we had database issues, it was the usual stuff: missing indexes, suboptimal config settings etc.

Could you explain what do you mean by "NFS hardware offload"?

Google search points mostly to VMWare-specific "VAAI NAS framework", which doesn't seem to be useful in case of database over NFS.

Some storage appliances have special hardware to accelerate NFS operations. Those may be able to come close to the same performance as a block storage device.

The other option with NFS is running it in software such as a NFS server on Linux configured with /etc/exports and backing the data to a block storage device. This is CPU intensive and would be much slower under load.

Do you find you are getting a dense enough # of pods per host on bare-metal?

That's an interesting question. Why would you anticipate a possible difference? Is the cluster running a different scheduler? If anything I would expect cloud-hosted managed clusters to possibly have less free resources due to running the pieces of the cloud provider's control plane.

There are bottlenecks in the Kubernetes architecture that start popping up once you cross about 250-300 Pods per node (the published limit in the K8S docs is actually 100).

This is typically more of an issue when running the cluster directly on large bare-metal nodes because in the cloud or on virtualization the host gets split into more smaller nodes from a K8S perspective.

Obviously this also depends on the characteristics of the workload, if you have a lot of monoliths that you have only barely containerized the Pod per Node limit may not be a concern at all even on bare-metal.

I don't recall any problems with pod placement, mostly because the machines are big enough to have ample room left over. We did move our ELK stack to a separate VM-backed k8s to be able to scale it better, but that was mostly because the storage box of the bare-metal cluster didn't have enough space to store more than a week's worth of logs.

Hey do you have any pointers on how to setup databases in kubernetes with nfs? Do you use stateful sets + the kubernetes nfs provisioner, or do you interface with the nfs store using a different storage provisioner?

Reply on news.ycombinator.com