Hacker News Reader

Comment by dharmab | Hacker News Reader

When we implemented Kubernetes at my employer, one of the things we made clear from the outset was that there is no LTS release of Kubernetes. Keeping k8s and friends up to date is at least one full time job (ideally a half-pizza team).

This also heavily influenced our choice of OS away from ones that emphasized LTS package management and towards more recent kernels with better BPF and kTLS support.

(We also intentionally used short-lived TLS certificates to prevent anyone from running a cluster longer than several months without updates. Worked great. There are workarounds of course but anyone aware of them is the kind of person who updates anyway.)

View on HN

That's why k8s is not a good fit for early startups. It will consume too much of your time that is better spent developing a product that the public wants.

imho if you're an early stage startup and you don't have an operations or devops person, you're better off either running everything off a single machine or doing devops the old way (as in baking machine images, updating launch configurations and doing service rollouts with via autoscaling groups).

Once you get more revenue, hiring an operations/devops person might be a good idea. A good operations/devops person might make you save his own salary in infrastructure optimization (depending on the scale of course).

another good pattern that i've seen in action is contracting the operations to an external company that's specialized in doing this. the approach would be to let the external company design and build the cluster for you, define responsibility boundaries for your company vs that company, and let such company take care of daily operations and cluster upgrades, possibly with on-call availability. A small support/consulting package (say, 15-20 hours/month) can get you very far.

We've been doing something similar with SigHup (sighup.io) and their Kubernetes Fury distribution, it's been working very well so far.

(disclaimer: i'm not employed by sighup, just pleased to work with them)

Agreed. That's exactly what I have done with a number of startups. In one or two weeks, they get a full CI/CD pipeline that allows them to be productive. All of them are very happy with the result.

It's so important to be deploying early on. Get the kinks out of the system and the code aligned for production.

If you hire a full time devops, they will most likely build too much infrastructure to keep themselves busy. And if you don't do any devops, the transition can be quite painful.

> baking machine images, updating launch configurations and doing service rollouts with via autoscaling groups).

I tried this approach, using Packer with Ansible to build the AMIs. It took 15-20 minutes to build an AMI. Would you suggest a different way of building the AMIs?

That was meant to be an example really, to aim at something less automatic and complex like kubernetes but still automated.

You might also consider using containers without kubernetes. Keep the same AMI, package your app as a docker container, make VMs download and run the latest release from a private docker registry on startup. Publish a new docker image tagged as latest.

Now you can rollout your autoscaling groups without running packer and ansible.

In that case use a managed cluster and it’s less work.

It has about the same issues but because it's managed and because you are a small company, nobody cares about your issues and you can do nothing about it.

zimbatm mentioned "early startups", I take that to assume you've just started the company and you're working on getting the product market fit right. There is absolutely no need for any clusters for that, unless you're anticipating Google-level traffic on launch day.

Most companies today are better off NOT running distributed anything, as everything will get harder to do, from logging to deployments. People start with clusters when they really should start with centralized services and only go distributed when they need to.

Indeed. A single machine is often more than enough for early startups. It's easy for developers to understand and debug.

What's important is to put the tooling in place to make the deployment reproducible. Deploy using Terraform and Docker to the single VM. Avoid modifying things manually. That's what provides the agility to then scale the infrastructure into bigger shapes as needed.

That's why we developed https://github.com/numtide/terraform-provider-linuxbox , to make that use-case even easier.

You still have to do upgrades every 3 months, sometimes with breaking changes.

Because the underlying infrastructure gets replaced, it means spinning up a new cluster, deploying the application on it to make sure that everything is working. Then switch the traffic to it. If the cluster hold any state this can be quite a bit of work to replicate all of the data.

Kubernetes clusters can be upgraded in place, though? GKE definitely allows you to do that.

It works most of the time. I have also seen multiple in-place upgrades ending up in weird places. Or some resource having changed API and we had to rush to upgrade the code.

What will you do if that happens?

It's a bit like upgrading Postgres. It's a good idea to at least make a backup before changing the underlying storage.

Yes, if you’re careful and do your research. Once you start extending Kubernetes you can back yourself into corners.

We’ve also found breaking bugs in Kubernetes that prevented in-place upgrades in some cases (which we reported upstream and were patched).

Running several production clusters on GKE, I can say that it is definitely not a full-time job to keep it up too date and healthy in that setting. I would imagine it's a completely different story on-prem.

> half-pizza team

Could you define this term? I tried Googling, but it looks like there are just two references and neither is clear to me.

2 or 3 people. It's a reference to Amazon's 2 pizza teams. The idea is your team is too big if 2 large pizzas isn't enough to feed everyone.

When you buy food for the meeting, how many pizzas do you need? If it's one pizza that's a one pizza team.

It's also a specifically American reference. A 1-pizza team in most of Europe, including Italy, would be a 2, max 3 person team. Often it would be a 1 person team :-)

Same in parts of America but for very different reasons.

In Italy a 1-pizza teams is absolutely a single person. We don't share pizzas :-)

Do you use managed Kubernetes or kubeadm?

I think if one uses a managed Kubernetes keeping it up-to-date is pretty easy. I did an AKS update in 10 minutes.

Neither. Both the managed offerings and kubeadm don’t meet our requirements. (Largely related to security, compliance, scale and multi-tenancy.)

What distro did you end up with?

CoreOS, then Flatcar.

Did you have a look at Fedora CoreOS ?

Reply on news.ycombinator.com