Hacker News Reader

Comment by majewsky | Hacker News Reader

Not OP, but my team is running on 100% k8s, mostly on bare metal. We do everything on k8s because of principle of least surprise: When apps are running as Deployments or StatefulSets, everyone knows how to deal with them (how to find logs, how to restart etc.) and it fits in the established deployment procedures (everything is a Helm chart etc.). When you have a single stateful service with a completely different configuration (e.g. RDBMS on a dedicated machine), it's always going to be "the odd-ball that only Greg knows how to upgrade". The whole point of k8s is not having these odd-balls. We have all databases in k8s, with physical volumes backed by an NFS-connected storage that's sitting in the same rack.

View on HN

> We have all databases in k8s, with physical volumes backed by an NFS-connected storage that's sitting in the same rack.

I'm surprised you haven't seen any major issues with doing this. NFS exercises some dark corners of Linux's VFS, and most people don't run databases on top of it (meaning you could run into a fair few pretty fun issues once you're doing enough transactions). From memory, most database vendors are pretty twitchy when you ask "do you support running $DB on NFS?".

> We have all databases in k8s, with physical volumes backed by an NFS-connected storage that's sitting in the same rack.

I was searching about this yesterday, how is your performance running a database in NFS? If your entire db can fit into the RAM cache it's probably fine, but when you have to do heavy IO is what I worry about with NFS.

I think it depends on your NFS. A storage appliance with NFS hardware offload will work much better than a software NFS.

Personally with bare metal database I'd try to do fibre channel if supported, then iscsi. Would only dabble with NFS if it had hardware offload, but even still I think block storage would have better iops.

AFAIK the reason for using NFS is that it's supported in k8s out of the box. I don't recall any performance issues for the things that we're doing. When we had database issues, it was the usual stuff: missing indexes, suboptimal config settings etc.

Could you explain what do you mean by "NFS hardware offload"?

Google search points mostly to VMWare-specific "VAAI NAS framework", which doesn't seem to be useful in case of database over NFS.

Some storage appliances have special hardware to accelerate NFS operations. Those may be able to come close to the same performance as a block storage device.

The other option with NFS is running it in software such as a NFS server on Linux configured with /etc/exports and backing the data to a block storage device. This is CPU intensive and would be much slower under load.

Do you find you are getting a dense enough # of pods per host on bare-metal?

That's an interesting question. Why would you anticipate a possible difference? Is the cluster running a different scheduler? If anything I would expect cloud-hosted managed clusters to possibly have less free resources due to running the pieces of the cloud provider's control plane.

There are bottlenecks in the Kubernetes architecture that start popping up once you cross about 250-300 Pods per node (the published limit in the K8S docs is actually 100).

This is typically more of an issue when running the cluster directly on large bare-metal nodes because in the cloud or on virtualization the host gets split into more smaller nodes from a K8S perspective.

Obviously this also depends on the characteristics of the workload, if you have a lot of monoliths that you have only barely containerized the Pod per Node limit may not be a concern at all even on bare-metal.

I don't recall any problems with pod placement, mostly because the machines are big enough to have ample room left over. We did move our ELK stack to a separate VM-backed k8s to be able to scale it better, but that was mostly because the storage box of the bare-metal cluster didn't have enough space to store more than a week's worth of logs.

Hey do you have any pointers on how to setup databases in kubernetes with nfs? Do you use stateful sets + the kubernetes nfs provisioner, or do you interface with the nfs store using a different storage provisioner?

Reply on news.ycombinator.com