Comment by m0xte | Hacker News Reader

Comment by m0xte | Hacker News Reader

Parent Full thread

It’s expensive and a nightmare of cost complexity which leads to an entire meta-industry of cost management which I’m sure is a net loss.

I’m really more concerned about the absolutely miserably shitty performance of snapshots myself which always goes unmentioned and unmaintained. So I’ve got a 4TiB volume online attached to a node which is blasting out fragmented writes galore (Jenkins - ugh).

So we need to back up this volume for DR. Snapshots right? Well not when it takes 28 hours to snapshot that volume. So off to rdiff-backup it is. Back in the last infrastructure cult of virtualisation this was a 20 minute turnaround.

Good job it’s 5 nines now. That means we only risk a 3 day recovery window 2000x less often. Actually that hurt writing that.

I’m sure someone will now immediately sell me EKS to run Jenkins now which is ten fresh layers of eye gouging pain between idea to delivering software, something which I barely remember being able to do efficiently at this point.

I was with you until >entire meta-industry of cost management which I’m sure is a net loss.

I am paid hourly and I have literally, in the past week, billed ~14 hours to save ~$150,000 per month. We work for a client and we strive to make sure we're not spending more money in labor than we're saving in AWS costs. I don't know exactly the rate I'm billed at that the client sees, but I know for sure that on aggregate, aside from the actual productive work that I know adds value, I have justified every penny I've been paid in the last ~2 years through my cost savings work alone.

Maybe some folks overcharge or undersave, but I take exception to your comment that you're "sure is a net loss."

Edit: our monthly AWS spend is in the 7 figures. We go through cycles where our priorities change, but when we're in cost savings mode, we regularly discover cost savings measure that are a) implementable in double digit hours and b) save six figures per month.

I’m not saying that it doesn’t result in a saving but that the fact that industry has to exist to keep costs constrained is a net loss.

Imagine if pricing was transparent, calculable and easy to rationalise the cost of a change. But it isn’t unless you pull a third party in.

Heh. I feel your pain. EKS is a nice platform, but I have seen the light and try not to run anything that requires a persistent file system on it. I feel like I'm shouting in the wilderness though- just adout every day is a "can't we deploy mysql here?"

Yes I’m a big proponent of stateless stuff. In fact 20 years ago I was crowing this to a deaf audience. Instead I find queues in databases that didn’t even need to exist. And our solutions, kubernetes for example, are complicated Rube Goldberg machines rather than things that make us sleep easy.

However really the software industry is all about state and where to keep it because that’s where the trolls collect their gold coins.

So what can we do?

I find gin helps to numb the pain...

So where would a queue go in a distributed system if not in some kind of database/file system? How do you handle resilience if the queue is not stored on disk somewhere? Paxos?

Well the point is more that a lot of stuff doesn't even need to be in a queue. It's just cookie cutter engineering. "I did this before and it worked so I'll do it again" without any consideration on what it does and why it does it.

Is this an EKS problem or a Kubernetes problem?

Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock.

Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much https://thehftguy.com/2016/11/01/docker-in-production-an-his...

Unless they are sharded and replicated distributed databases...

> Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock.

Really? Can you share details? I'm asking because I know people who run databases in Docker for years and have no issues. Just store data on a volume.

> Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much

Maybe not, haven't had a storage issue with Docker myself since maybe 2017. That article feels like a big rant but yes, it's ... old.

Docker volumes are great. I've also run databases in docker without issue. But they're different from kubernetes "persistent volumes" and "persistent volume claims"

The issues come when you move into high availability (specifically in a Kubernetes context). K8s may decide to move your database to a different node (server) on a whim, and suddenly your data is gone. You have to solve this by having mounted, block-level storage available to your cluster (eg AWS's EBS; the product being discussed) so you can use persistent volumes. That comes with added complexity, and usually the need to incorporate db operators into your stack.

As a result, a lot of k8s admins (myself included) choose to pay crazy amounts for RDS and S3 to avoid the complexities of persistent data.

I'm just learning all this; but couldn't you run a DB Cluster persistently on EC2, managed separately from K8s, and VPC it to the K8s cluster? Is there transfer charges involved and/or is the latency to night to consider a setup like this? Is this cost effective to RDS?

If you have a DB cluster running on multiple EC2 instances, you can definitely do that. There are advantages/disadvantages compared to just paying for RDS (eg managing backups, system complexity, and failover) but if you already have that expertise in-house, I can imagine a scale where there might be cost savings.

My comment was specifically discussing the challenges of keeping block storage-dependent, stateful apps within k8s (as opposed to using RDS or rolling your own HA cluster on EC2 or elsewhere). Apologies to GP if I incorrectly assumed they were discussing container volumes more broadly.

Yes, hence „volume”, not „docker volume”. For k8s, always a PV. rexray for mesos.

That article is about broken container filesystems, as in, the root filesystem of the container, but that filesystem is not where you would ever store the data of a database. In kubernetes/eks you would use a PersistentVolumeClaim with an ext4 or xfs filesystem. This would not share any of the overlay filesystem problems and StatefulSets are quite good at managing these.

We run over 900 very stateful MySQL database clusters and several hundred zookeeper clusters in kubernetes on ec2 with ebs successfully. Sure, we've hit our share of problems over the years, but storage itself is solid.

Indeed. The reliability of the system depends on the reliability of the block storage. There are many flaky solutions and many good solutions, and they are unrelated as to whether or not you are using containers.

Meanwhile arguably the whole reason Kubernetes took off compared to other container orchestrators is support for StatefulSets and Persistent Volumes.

Lots of people run their databases on Kubernetes, but it’s best to remember that Pod ephemerality makes it a more complicated orchestrator than them on VMs. That’s said, the market seems to have decided that’s what they want.

Funny enough, I agree and often hear of statefulset as a pro of kubernetes, yet I've much more rarely seen companies actually using the statefulset let alone migrate their databases to kubernetes.

There is a gap between liking a technology and migrating a hundred critical production databases onto it.

These things take time. Having sold a stateless container PaaS for many years, the main technical objection is they couldn’t run a database on it, you still needed VMs (even if they were on demand).

People caught in tech fads get obsessed with mechanism over outcome.

Reply on news.ycombinator.com