Hacker News Reader

Comment by chrisbolt | Hacker News Reader

Is this an EKS problem or a Kubernetes problem?

Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock.

Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much https://thehftguy.com/2016/11/01/docker-in-production-an-his...

Unless they are sharded and replicated distributed databases...

> Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock.

Really? Can you share details? I'm asking because I know people who run databases in Docker for years and have no issues. Just store data on a volume.

> Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much

Maybe not, haven't had a storage issue with Docker myself since maybe 2017. That article feels like a big rant but yes, it's ... old.

Docker volumes are great. I've also run databases in docker without issue. But they're different from kubernetes "persistent volumes" and "persistent volume claims"

The issues come when you move into high availability (specifically in a Kubernetes context). K8s may decide to move your database to a different node (server) on a whim, and suddenly your data is gone. You have to solve this by having mounted, block-level storage available to your cluster (eg AWS's EBS; the product being discussed) so you can use persistent volumes. That comes with added complexity, and usually the need to incorporate db operators into your stack.

As a result, a lot of k8s admins (myself included) choose to pay crazy amounts for RDS and S3 to avoid the complexities of persistent data.

I'm just learning all this; but couldn't you run a DB Cluster persistently on EC2, managed separately from K8s, and VPC it to the K8s cluster? Is there transfer charges involved and/or is the latency to night to consider a setup like this? Is this cost effective to RDS?

If you have a DB cluster running on multiple EC2 instances, you can definitely do that. There are advantages/disadvantages compared to just paying for RDS (eg managing backups, system complexity, and failover) but if you already have that expertise in-house, I can imagine a scale where there might be cost savings.

My comment was specifically discussing the challenges of keeping block storage-dependent, stateful apps within k8s (as opposed to using RDS or rolling your own HA cluster on EC2 or elsewhere). Apologies to GP if I incorrectly assumed they were discussing container volumes more broadly.

Yes, hence „volume”, not „docker volume”. For k8s, always a PV. rexray for mesos.

That article is about broken container filesystems, as in, the root filesystem of the container, but that filesystem is not where you would ever store the data of a database. In kubernetes/eks you would use a PersistentVolumeClaim with an ext4 or xfs filesystem. This would not share any of the overlay filesystem problems and StatefulSets are quite good at managing these.

We run over 900 very stateful MySQL database clusters and several hundred zookeeper clusters in kubernetes on ec2 with ebs successfully. Sure, we've hit our share of problems over the years, but storage itself is solid.

Indeed. The reliability of the system depends on the reliability of the block storage. There are many flaky solutions and many good solutions, and they are unrelated as to whether or not you are using containers.

Meanwhile arguably the whole reason Kubernetes took off compared to other container orchestrators is support for StatefulSets and Persistent Volumes.

Lots of people run their databases on Kubernetes, but it’s best to remember that Pod ephemerality makes it a more complicated orchestrator than them on VMs. That’s said, the market seems to have decided that’s what they want.

Funny enough, I agree and often hear of statefulset as a pro of kubernetes, yet I've much more rarely seen companies actually using the statefulset let alone migrate their databases to kubernetes.

There is a gap between liking a technology and migrating a hundred critical production databases onto it.

These things take time. Having sold a stateless container PaaS for many years, the main technical objection is they couldn’t run a database on it, you still needed VMs (even if they were on demand).

People caught in tech fads get obsessed with mechanism over outcome.

Reply on news.ycombinator.com