We cut costs by 70% by moving from GCP and CockroachDB to Hetzner and PostgreSQL | Hacker News Reader

We cut costs by 70% by moving from GCP and CockroachDB to Hetzner and PostgreSQL

blog.mikoto.io

43 pointsCactusBlue69 comments

It's wild to me that we are talking about a site running on a single 2 core VPS costing $15/mo and the developer didn't stop to consider that maybe they don't need Kubernetes for this. Like...just run your damn services on the box like we've been doing since web hosting first became a thing.

For me, the main benefit of Kubernetes is declarative deployments. You can have the entire deployment automated through this and GitHub actions, without much fiddling with the production environment manually. I know, it's overkill and I don't like it that much either, but it's the industry standard.

> but it's the industry standard.

I strongly disagree with this statement. Even at the most bubbly of Silicon Valley events you will not see kubernetes for a smaller deployment be an industry standard.

It’s ok to make architectural mistakes, or otherwise over engineer a thing. We all do it. Let’s not standardize it.

And to your credit thank you for publishing this example as a learning opportunity to others.

I mean as in what the big (and small) clouds support. I haven't seen any clouds supporting Hashicorp Nomad or Docker Swarm!

You shoudl be able to run any of those things on any cloud. A box with sshd is a box with sshd.

k3s is a teensy sqlite based orchestrator, something tons of devs run on their laptops for local dev. I think those are all single box clusters, why is it crazy to run it on a server but not a laptop?

You can get declarative deployments via a systemd config. If you really want to use Docker then use a Compose file.

If jobs are what you are concerned about then I can assure you no company is skipping on software developers because they don't know Kubernetes syntax.

imo systemd is more complicated than k3s

That doesn't capture the same thing. It doesn't capture volumes, versions of binaries, nginx/postgres/whatever configs, etc.

Ansible/Salt/Puppet/Chef/CFEngine are closer comparisons.

With Docker Compose you can define volumes, versions, and configs can be volumes that point to a file that lives alongside your compose.yaml.

Sure, depending on how you want to draw a line around features. Compose won’t do multi-node orchestration of deploys (I.e. update backends 1-3 sequentially, and revert the change on all of them if any fail to deploy).

It also doesn’t manage your configs; ie if you need to roll back, it won’t do that for you (though git will let you do that pretty easily).

I think Compose is great for the niches it serves; I have a lot of stuff that still runs on Compose because it doesn’t need any k8s stuff yet. I just don’t think it can be honestly called a k8s replacement. Compose offers equivalent features for some kinds of apps in certain situations, but k8s is a broader tool by far (for better or worse).

Agreed—people on here are only recommending compose because OP only has the single node, which places their app squarely within the realm of what compose is very good at.

For this, you can use nix with deploy-rs, and define those configurations declaratively.

Overkill is an understatement and “industry standard” is a misrepresentation. There are PLENTY of more appropriate options here, for this scale, without going off the beaten track. This is wanting to use the big kids’ tools when it doesn’t make sense to.

Use docker compose or ansible for this scenario.

Those seem like different things, I thought Ansible was more for configuring individual boxes.. then you would use something like Helm to configure Kubernetes.

Tech stack at my last workplace was insane like that. To deploy, they woud have a Buildkite job would launch Terraform, which would then be configured by Ansible, then k8s would deploy on the cluster. I think they could have avoided all this ess k8s and probably Terraform and gotten away with one big box and shell scripts (which they had plenty of anyway), and used swarm/compose, but what do I know.

If I have to choose I’d rather learn k8s than ansible.

For a tiny deployment like this, Docker Compose gets you declarative deployments in much the same way with arguably a much smaller learning curve and fewer moving parts.

I self-host ~10 services on a single physical server in my home, defined in a series of docker-compose.yaml files stored in a git repo. It's trivial to set up, declarative, and easier to run locally than k8s.

It’s a bit weird to me to mention compose and kubernetes without bringing up Docker Swarm which feels like a nice middle ground. We actually run it at my very small SaaS company and it seems like a good fit for us though it’s still relatively early days for us; it’s been a very slow transition.

I've never used Swarm myself, so I don't recommend it. I've heard several people like you, though, who found it works.

Obviously not needed for a single node, though.

A different angle -- Ansible and many others provide declarative deployments too.

No, … Ansible is not declarative. Nowhere near the same as a k8s workload.

And I know they say that, but it's more or less YAML-as-a-programming language. It is pretty trivial to have side-effects that will persist well beyond and outside an Ansible playbook.

It's certainly better than a complete snowflake of a VM, but it's really closer to saying "we init this VM with this shell script."

Promotion driven development.

Fwiw I’ve deployed services that could handle 30-50k rps on 1vcpu and 1gb ram. Like a couple bucks a month. It used ssd storage and only hit a main rdbms when there was a cache miss.

When I hear stories like this I’m reminded of how dumb and inefficient most organizations are. My colleagues would over engineer stuff so much and then leave a huge bill and mess behind. The primary reason was so their clueless eng manager would consider promoting them.

Honestly if I was gonna set up a CI (or some other system that wants to add and remove a lot of containers dynamically) on my home server I might consider adding k3s so it’s easier to install / manage / monitor.

For a single node cluster, just use minikube (or RKE2 if you have masochistic tendencies). And if everything runs on a single small vps, why even use postgres and not SQLite.

k3s (512mb ram suggested) is smaller than minikube (2gb ram suggested)

I've said it a few times before: k8s is such a distraction for so many startups I've been in (~5). All of them fall into the same trap: write entire arch around k8s and then not really invest the engineering into building it out fully. Instead they all seemed to settle on "hey this works" which kicks off the technical debt receipt printing.

In all of them fixing things often went like this... * A: XYZ is broke. * B: OK, is the pod running? * A: Yeah. * B: Did you try killing it? That usually fixes the problem.

It's madness k8s is generally always the root cause of the problem because of resource constraints, node issues, a misunderstanding of H/V scaling, and the list seems to just go on.

At this point I feel like k8s is the c4r (cancer) of a lot of startups that would be better off just running in a more traditional VM / container strategy and dealing with growing pains if and when they become apparent. Also, it's almost guaranteed to cost less. Kaa$ is a racket.

Many startups would also be fine with some serverless container solution (AWS Fargate, Google Cloud Run, etc.) It doesn't fit all applications, but when it does, it's generally less overhead than K8s, both operationally and $ wise.

Let's put k8s aside. If I had a complex system needing say five boxes, I might be able to admin it well - startup, shutdown, upgrade - if there was a simple orchestrated glue between the boxes e.g. on the control box 1, run a script to start daemons/pids on boxes 2, 5 which blocks until success or failure ... something more sophisticated than remsh/ssh. This is old school, but it's just a script or three.

As an SRE who’s moved between a few companies recently, I’ve noticed a trend towards simpler infrastructure options rather than k8s. Most places I interviewed at used AWS with just fleets of EC2s managed through ASG and instance refresh.

Deployments are no downtime. It’s pretty nice and if you want to run with docker on the host instances you can but you can also just install the application using systemd.

This is what we do for GCE. Our CI builds a system image and then that's deployed across our datacentres. Worked perfectly for years, but still people want to switch to kubernetes. What can you do?

Oh definitely that happens and from my experience I don’t think there’s a lot that can be done unless there’s strong faith in engineering leadership to push back on the complexity.

How do you build ec2's? Funny thing is, everyone will have their own answer. Few will look alike. It's maybe simpler, but also bespoke.

There's different definitions of simpler. I've rarely felt like folk's have a great grasp when they cobble together a bunch of options. Things that maybe one engineer started as a simple project 5 years ago grow and sprawl. They rarely stay simple.

What this team gets out of the box here is not to be trivialized. You havent started talking to the ci/cd that ArgoCD deploys. How are load balancers setup & managed, and at what price (Traeffik here is basically free)? Well managed Postgres with backups & replication, Lets Encrypt... handled. Is it really simpler to keep DIY'ing bits and bobs? Are you confident your simple start will work reliably & won't need improvement? What is better about your pile "simple" bespoke solutions, and why not have a platform that integrates varying concerns with consistent management patterns & autonomic operators?

EC2s are provisioned with Terraform or if you want to bypass that, click-ops. Nothing custom is needed. You can define a small shell script that is known as userdata that runs when the EC2 boots. That's standard and used everywhere that EC2s are used.

If you want more customization - again not necessary - you can build via Packer. The thing is, you have to do that anyways with Docker or whatever your container runtime happens to be. Docker images need patches as well if you install dependencies like openssl, libpql, etc.

ArgoCD - ArgoCD is great but it's something that has to be separate managed and comes with a database, UI layer, etc. and it's own set of permissions. The whole point of ArgoCD is to have deployments specifically around GitOps for container-based applications. Unnecessary if you don't have K8s.

Load balancers are simpler with ALBs and route directly to the auto-scaling groups. Traeffik is something that you need to manage. Also, K8s typically requires the use of external load balancers to work in the cloud as it is which means there's a layer of complexity that was introduced. Typically, K8s through the External Load Balancer Controller will provision the LB and attach targets. So it adds complexity. Traeffik isn't free, you are paying for the compute in your cluster which has the overhead of the control plane.

Let's Encrypt is another layer of complexity that has to be managed. AWS has ACM which can dynamically create and renew your certifications without you having to do anything.

K8s as a cluster needs to be managed and introduces significant breaking API changes across minor versions (1.23-1.24, etc.). Postgres managed as a stateful application also breaks some of the benefits of K8s in that it needs to be managed as a StatefulSet rather than as a ReplicaSet which means you likely need different deployment mechanisms to avoid destroying it accidentally.

Ironically, this architecture doesn't need any customizations at all. You can provision it with a few modules in Terraform or you can click ops the entire thing. K8s on the other hand, requires pretty constant attention. You've now introduced the management of the cluster, the building of the image, management of ArgoCD, Traeffik as well as external load balancers.

Basically: - Route53 for domain management

- ACM for cert management

- EC2s for LBs

- ASGs to manage EC2s

- Small shell script within userdata to configure the host vm

- RDS if you need a database

Avoiding the operational and actual costs of K8s.

I think that there are compelling reasons to use K8s:

- Lots of smaller services that need to interface with each other

- Product is deployed to multiple clouds or in a hybrid setting

- Infrastructure team has deep expertise in K8s

I think you'd spend longer just figuring out terraform, step 1, than you would doing every single piece of this blog post.

Yep. What you describe is how we were doing things 10 years ago!

The time investment here doesn’t really make sense to me. That 70% savings was actually only $33 a month.

Was it really worth spending that dev time and effort for such a small return rather than the product itself?

And yea now you get to manage your own cluster… thats an ongoing cost too, not free.

Yeah feels a bit disingenuous to title the article “70%”. While correct, it elicits the thought of a much more interesting migration than what actually happened.

This is what I always think. People will spend $10,000 on engineering time (2 engineers working ~1.5 weeks) to save $500, that then turns into a COST later because they have to spend more on maintenance.

Now they get to write an article about it.

48.08 USD per month on a single-node GKE cluster w/ 2 cores 4 gigs -> 15.34 USD per month Hetzner VPS w/ 4 cores 8 gigs.

48 seems really low to me, I've never seen a k8s cloud cluster for less then 100-150 a month on the very low end, also doesn't take egress into account

how do you even run a k8s cluster "on 2 cores 4 gigs", don't you need at least 3 nodes?

You need 3 nodes for a high availability control plane if you’re managing the Kubernetes control plane yourself.

With GKE, the control plane is totally managed by Google Cloud, things like etcd etc are not running on any of your instances.

It’s fine to have just 1 compute node (like the before state), or to put all the k8s control plane on a single node (like the after state) if you don’t need high availability.

interesting thanks, do you know if AWS offers something like that as well?

I think the comparable product is AWS EKS (Elastic Kubernetes Service)

It would have to be a zonal cluster, I think. A regional cluster would have 3 nodes, minimum (ish¹).

(Unless maybe if you can do a two-zone regional cluster? But I'm not sure why you'd ever do that, though. I don't have much experience w/ the zonal ones, since typically we want our service spread across multiple failure domains. OP seems to be in the bootstrap phase, so the cost savings might be > the reliability at their present scale, to them, which I can respect.)

¹you can have a zero node regional cluster, but it isn't going to do much work.

If your app needs 2 cores and 4 gigs RAM, why are you running it on a cluster in the first place? That "cluster" is going to have more resources going to overhead/management than running the application itself.

I should write a blog about how I saved the same amount in absolute terms by turning off the stuff i forgot is running in the cloud.

Side note: the anime girls all over is kind strange. Like the header is giant cartoon tits? Why?

I use AI-generated anime girls for article covers because I'm a weeb and it makes the rather-thick wall of text slightly more bearable.

Yea, the obivously AI-generated big-titty anime woman header was super out of place.

Seems strange to have used CockroachDB but not leverage any of the distributed nature of it? I was looking forward to some interesting database problems in this article, but it was quite anticlimactic.

There were also some performance issues regarding the read/writes compared to plain PostgreSQL, but these were relatively minor compared to issues with the extensions and tooling

I hear that you and your band have sold your guitars and bought turntables. I hear that you and your band have sold your turntables and bought guitars.

I want to make a Yaz record

Fun article! I had a great time playing with k3s a while back and this makes me want to get back to it

I predict hard times for GCP as they will get squeezed by Amazon and Azure, plus the kind of belt-tightening we see here.

GCP is expensive even compared to Amazon and Azure. The only reason I really used it was because it was the cloud that I was used to (when employer was covering for the cost), and I kinda stuck with it.

I predict a very hard time to CTOs once shareholders realize how money they are wasting on the cloud.

hetzner egress is x100 cheaper. it’s a good reason to switch.

Not really.

You didn't factor in the time it takes to manage k3s yourself versus GKE which is managed.

Maybe it is a very simple setuo

Probably. But it was only a single command to get up a k3s node - not much difference in effort for a relatively low-workload, low-critical service.

It's probably only a few clicks to get a GKE cluster.

The point is that it's not really a fair comparison.

Self management could be expensive if it’s actually going to burn hours of dev time.

At teensy scale is k3s really gonna take much “management” after the bring-up? I don’t think so.

For a teensy team, it seems totally worth it to take a runway from 12 months to 40 months for the price of a few afternoon’s work.

This. Just upgrading this made me (a fresh university grad with minimum savings) basically and my startup "default alive" so that the startup can be bootstrapped

The wins here are absurd, & the resistance to them remains as staunchly resolutely inconsiderate as ever.

I've done Lets Encrypt by hand. I've had rotation not work as expected. I've done postgres by hand. I've spent work weeks setting up and testing backups. I've dealt with fallout from another person's backups scripts not working. I've worked with teams that took weeks to setup replicas. I've been on teams that have spent quarters building and improving ci/cd systems.

Folks will never ever shut up about how you might not need Kubernetes, but damn this post documented an extremely short path through a couple kind of annoying to DIY things that just bloody work and at a much much much higher level of productization than the bullshit bespoke scripts your own team would put together.

Traeffik load balancer with l7 routing? Already running. Let's Encrypt? Works, reliably. Internal DNS? Out of the box. Postgres? Incredibly well managed. Ci/cd? A nearly Heroku like beautiful world of gitops. These pieces all snap in to the general framing with ease & grace; this whole post is a complete DIY infrastructure stack that one could do in an afternoon, and even better the output is predictable & understandable & legible by another practitioner, which the "real men just do it themselves" attitude doesn't cover.

The bullshit asymetry principle feels soooo strong here. Spreading fear uncertainty & doubt, yucking people's yum, is going to waste an incredible amount of time.

Dont you have it out of the box with some distros?

Thanks for the Japanese cartoon porn warning!

Kubernetes - is that thing still around?

Reply on news.ycombinator.com