You shouldn't use Kubernetes as a startup

I've spent a lot of time over the years trying to think up the perfect analogy for software development and I've never quite found one.

It seems like they all tend to break down but cooking is ok for some parts.

To that end I see these conversations as

"never use stainless steel to cook a hamburger"

"never use cast iron on a food truck"

"Always use an crock pot for soup"

Especially for startups it just doesn't matter that much. Use whatever you know how to cook with. If your DevOps guy can deploy kubernetes into AWS managed with GitOps designed by GitHub copilot and ChatGPT in terraform in 40 minutes and manage without waking up in the middle of the night...that's fine.

Or if you have a die hard linux admin who wants to buy servers on eBay and host them in the basement and do hardware updates at 2am...that's also really fine.

Stackoverflow literally uses windows servers.

These are all just different methods of cooking. Sure we each kind of prefer our way and insist it makes the food come out best, and sure there is truth to that at a point - infra does eventually wind up mattering.

But mostly we're just nerds arguing about what brand of chef's knife we use and that sort of thing.

Reasonable-ish counter concerns but way overinflated in each case & loses it's persuasiveness by trying overhard to sell fear.

Plenty of people run their etcd on their control plane. If you are indeed a startup you might even consider not having dedicated control plane nodes. The idea of side-cars taking up another 100% more resources is perposterous, absurd. Running ingress takes up some overhead, but it also allows you to consider running a wholeot less load balancers if you have multiple services, letting kube do all the routing; this can save a good chunk of change on infra costs if you go for it.

Things like errors devops, ci/cd are presented in the scariest possible terms. But it's not like these are non-issues without kubernetes. I've seen man-year of time go by building AWS Beanstalk and AWS ECS scripts, just to deploy stuff (build wasn't any better!). There's such a universe of mature & solid tools for Kubernetes that encompass a much more end-to-end handling. Plenty of gitops starter kits. Many of these you can try out in a day or less, see if you like the look & feel. And abandon and swap-out just as quickly. There's a much better base of options here than there is anywhere else.

Articles like this always sound the same: That standard tool is too complex - you should roll your own home-brewed solution with no docs instead!

No thank you. Standardization brings its own simplification, even if the one standard is more "complex" than a hodgepodge of non-standard tools.

"If you add Kubernetes to this setup you will need to make your machines double the size". What? No.

"Kubernetes is not stable". This article is ridiculous.

There is a saying in Kubernetes "containers are cattle, not pets". That is from a software perspective, the actual saying should be "hardware is cattle, not pets, and your containers are fleas on the cattle's back". If you aren't running on enough hardware for servers to be cattle not pets, then Kubernetes is solving a problem you don't have, and probably won't have for a long time.

While I mostly agree with the implied sentiment of "you probably are better off without k8s", this article makes no persuasive (or at least not objective) arguments to that end. I worry a bit for the HN flaming that is likely imminent here.

Suggestion for OP: expand on your ideas with concrete examples instead of hand-waving in the abstract. There's plenty you can cherry pick from (https://k8s.af has a bunch)

e.g. You say at one point

> "It makes perfect sense with all the network encapsulation over the cloud network virtualization over the real network well things tend to get crazy"

I'm not sure what that sentence is meant to convey. k8s networking is indeed complex, but what is "things tend to get crazy" supposed to mean? It would be helpful if you provided examples for what you seem to treat as obvious for the rest of us.

If you know kubernetes, it lets you punch _way_ above your weight in terms of running lean.

This and all the other similar posts that way overconfidently state "you don't need k8s" seem to miss the most important factor IMHO: Do you know it or would you have to learn it?

If you have already been using K8s and you build a side project or launch a new startup, then K8s is probably a great choice for you even if you only have one Deployment and Ingress route. I can take a new app (assuming it already runs in docker) and have it running in a brand new k8s cluster is 20 minutes. I could have it running on a static ec2 instance/linode/droplet/whatever in 30 minutes, but then you have to solve a ton of problems yourself such as CI/CD and deployment automation, etc that k8s gives you for free. K8s is a great choice if you already know how to use it.

This also seems to miss that for startups the best path to k8s is a managed k8s solution. Linode (for example) has one that is wonderfully easy and affordable, and will scale with you a long way.

Lastly, as a person making decisions on tech to use, you are investing in your own future as well. Learning k8s is one of the most useful skill I think a dev can do, behind things like learning bash and standard *nix tools like grep/sed/awk.

Edit (and self-plug): speaking of awk, if you're interested in learning it I put together a free "course", originally a talk I gave at Linux Fest Northwest on awk that I've gotten a lot of positive feedback for. Github: https://github.com/FreedomBen/awk-hack-the-planet Youtube: https://youtu.be/43BNFcOdBlY

What a load of rubbish. If you already know k8s then you pick a managed cluster like GKE has on offer and everything is set up and up and running in less than a day for less than $200/month.

Good call out. There are a lot of unnecessary complexity plaguing the tech scene.

Scalability, microservice, SRE, SPA, unit testing, design system, Restful/graphQL everything. Not saying there isn't a place for them, there is. But probably not at day 1 when the thing you intended to make your living didn't even exist.

I manage an engineering team at one of the big 3 cloud providers advising startups on their cloud deployments. We often get earlier stage startups asking us for help with building their deployments on Kubernetes. We often suggest they consider a simpler solution, so that they can focus on building their product and not worry about managing k8s. Some of them do. Most stick with k8s. Rather than assume that they all don't know what they're doing, I tried to take their point-of-view and understand why they're making this choice. What I learnt makes a lot of sense:

1. You won't get fired for choosing K8S. Tech leads of startups, just like anyone in a position of responsibility, need to be a bit conservative with their choices. As CTO, if something went wrong with your deployments, or you're having scaling issues, the last thing you want is to have to explain to your CEO or your investors why you've chosen a less popular and recognisable product.

2. Startups are conscious about vendor lock-in (or any kind of lock-in). That makes a lot of sense - early in the days of a startup you don't really know what direction it will take, so you want to keep your options open. Make a choice that limits your freedom later down the road and it could be fatal to your startups as it grows and adapts. It's reasonable to agree to pay a price for this flexibility.

3. Hiring good engineers is always hard, and you don't want to make it harder for yourself by choosing technologies that are less familiar. A common objection would be that basic technologies like containers, VMs, and serverless functions are easy to adjust to for most engineers, but as a startup you really need to optimise for the common denominator or you risk wasting time on onboarding engineers on tech they are not familiar with.

4. Startups often don't spend as much time and energy on familiarising themselves with every new technological development and option. They are busy building a product. It could be that there's another, better option for them, other than the most recognised brand in the industry, but if they spent their time on learning and evaluating all these options that's time they wouldn't spend building and shipping features to their users.

5. Startups are optimistic by nature. They have to be. Tell them that they don't need the complexity of K8S at their current scale and they'll just remind you that they're aiming for ∞X, any day now. Most of them will not reach even 10X any time soon, but they have the right attitude to behave as if they will.

I disagree with most of the claims presented here as facts. I believe that they are true experience reports for some companies, but I don’t think they generalize as strongly as claimed.

No, k8s doesn’t double the number of machines you need. You don’t need pods running more than what would be on a dedicated node.

No, you don’t need a full time engineer to run your cluster.

No, k8s isn’t unstable, I have used it in prod since v1.5 and have only seen a small handful of issues. Recent releases are boring.

I moved my startup onto k8s about 10 months in, and was our only infra engineer through series B raise @10 engineers. It took a small fraction of one expert’s time. This was widely regarded as a great idea by everyone who came later.

Maybe if you don’t have a clue how k8s works it’s going to cost that much time.

And there is the key advice where I probably agree with the author on most situational recommendations (while disagreeing on the generalization): don’t learn k8s to deploy at an early stage company. But, if you know it well already, it’s a fine choice that is unlikely to hurt short-term and might help a lot later.

Things you get: infra-as-code, with a (subjectively) way nicer API than Terraform, that you can plausibly run and test on your local machine. Easy scaling setup for unexpected growth spikes. Review apps within reach if you need them. Ditto GitOps. Managed infra because you are using GKE or EKS like a sane person (rarely need to think about patching your nodes; GKE patched the Spectre/Meltdown vulnerable for me weeks before the public announcement). Flexibility to easily run any new service (say, you want to spin up a Jupyter Hub cluster).

There are so many infra options that work for small companies that don’t have real infra challenges; if you don’t know one well already, just use Fly/App Engine until that gets too expensive. Or a single VM running a docker container; that gets you surprisingly far.

Meta: for a startup particularly pre-PMF, you should minimize the amount of learning you are doing and just iterate as fast as possible with the stack you know best. Even post-PMF this holds for quite some time until you start to hit cost/scale pain.

It depends upon what your startup is doing. Maybe Kubernetes is overkill for a basic "CRUD" web app backed by a relational database and some object storage.

It's very helpful at our business that runs a variety of web services that are powered by batch data workflows. Kubernetes enables us to spread both the process workloads and the batch workloads across sets of servers, with redundancy.

For others in a similar situation, I'd highly recommend the combo of Kubernetes and Argo Workflows. We wrote a blog post about one of our workflows at https://www.interline.io/blog/scaling-openstreetmap-data-wor...

"I saw people moving from, the following tools have these benefits and these drawbacks, to it is trendy so lets do it."

What is that supposed to mean? I was interested in reading, but when the very first sentence is impossible to understand it promises a painful read.

No offense intended[0] but if it takes 6 hours to debug a coredns issue why would it take less to debug a similar issue with a different dns?

I can get down with not using operators. If you're unfamiliar with k8s, adding another layer of something you're unfamiliar with makes little sense. But regular boring k8s is pretty straightforward stuff that often drives existing, established tools like ip/nftables, nginx, etc.

0: seriously, I've spent what I felt was excessive time trying to resolve bugs

If you have complex orchestration needs, then none of these five criticisms apply, as any one-off solution would be even worse. If you don't have complex orchestration needs, then there's no need to use k8s.

> Lets consider you are a small to medium size startup your application consists of a couple of backend services a database a caching service and a load balancer.

This is very squarely on the "you don't have complex orchestration needs" side of the spectrum.

This sort of article is spot on for lots of standard CRUD apps, but doesn't mention the fact that there are plenty of cases where k8s (or similar) really would be a big benefit. If your application requires a lot of scaling up and down to be viable for instance, k8s can be a good fit. I think a better article title would probably be "you shouldn't default to using kubernetes as a startup".

How do I get continuous deployments without k8s? I value being able to deploy new code quickly, easily and fearless, especially in high speed low drag environments like startups. I don't see how I can do GitOps like workflows with any other tools that have the same large mindshare and breath of commercial, managed offerings.

As a start-up*, you shouldn't use Kubernetes managed by GKE, EKS, etc; you should use bare metal Kubernetes: "server rack" somewhere near your desk, 3 virtualized nodes with Ubuntu, MetalLB as load balancer, local-storage StorageClass with cron backups on another machine, Prometheus with Grafana for monitoring, CertManager DNS01 with Let's Encrypt for SSL, Contour/Traefik for routing, self host Docker registry, self host CI, Minio for object storage, Mongo/Postgres/etc. for database, Redis for caching. First setup time: 1-2 days, maybe one week to work out the kinks. Initial cost: ~$500 for a refurbished Dell/HP server, or go extreme with 3 RaspberryPis. Monthly cost: local electricity price, internet monthly fee. Current uptime with the above setup: 139 days (had a power outage 139 days ago, previous uptime was around 200 days).

* start-up: endeavour started with negative, $0, or close to $0 in bank counting time to market in years; not a start-up (maybe a start-middle): company with millions in VC money hoping for time to market yesterday.

How does they author (or anyone else) propose that you do autoscaling without locking yourself into a single cloud’s apis? K8 is not something we want to invest in but it seems like the only cross-platform game in town, (which we want to take advantage of gpu prices and credits)

For all the people who shared, I am humbled and thankful. I am trying to make it sane for small business to work with the currently quickly evolved tech stack this is a part of mindset I am trying to reduce costs and stay agile and I am doing that in my own business.

I agree with most of this except the part at the end that advocates "normal hosting".

Don't do that either. Just go all in on a hyperscale cloud's serverless offering.

No patching, no servers, no upgrades, no networking, so much easier to manage than even a single VPS.

I completely agree, there are enough PaaS out there that run containers, that are much better starting out points, that is until you need and can afford all the intricacies of k8s.

Basically just an argument against premature optimization. Which is fine. In my experience, small startups tend to use whatever arch. they think is fun + gets spun up easily.

You can go beyond Kubernetes into serverless, but other than those two, your tech department would be deviating from industry standards and paddling upstream by hand.

I think https://doyouneedkubernetes.com/ said it better...

TLDR: Because if you did, you will likely blow through that VC cash you were given, then you'll ask for yet another round of funding.

Maybe they will throw more money at you after blowing it up on K8s operational costs for your startup. But what if they say no this time? That is always a possibility and even today is a high likelihood than previous years.

Perhaps not using K8s as a startup might save your startup in speed-running into bankruptcy, especially when you are not making any money and chasing free users for years.

> If you add Kubernetes to this setup you will need to make your machines double the size just the extra side-cars that you need make things functional add 3 machines to be the control plane of Kubernetes, add 3 more machines for etcd cluster, add multiple internal services like ingress controllers and log and metrics collection.

No, you really don't. I run multiple 3-node (i.e., the bare minimum to survive an AZ loss) clusters, and they are just fine. We run multiple workloads on them. It's three entire VMs of RAM and compute. If that's not sufficient, you need to dig into which of the services you're running on it are eating too much of whatever resource is falling short, and conclude through debugging whether that's a bug, or you need to scale up. That's the exact same procedure as a service on a VMs.

If your counter-example is not doing "log and metrics collection" — that's an apples-to-oranges comparison. I run both k8s & raw VMs, and … both run these "side-cars". Not only do both run it, they're often the exact same process.

Even the ingress: in most "service is deployed to VM" scenarios I've seen, often the service is fronted with nginx acting as a reverse proxy. That's not only fulfilling the same role as something like ingress-nginx, it's practically the same process.

Most startups should probably go with a managed k8s, in which case they're likely not going to even have etcd in the cluster, nor more of the rest of the control plane. (IMO: don't choose AKS, but IMO the problem there is the A, not the K.)

> You already are paying double of what you would have if you just excluded Kubernetes out of this because in a medium to small business there isn’t any feature of Kubernetes you are really using.

You shouldn't be, because again, I think the k8s example is grossly over-provisioned.

But I also think k8s's ability to move workloads between nodes easily lends it an advantage here: if I need to change the underlying makeup of the VMs, I can. That means that I can shrink-to-fit them. In my career eng spinning up VMs are far more likely to over-provision than to right-size or under-provision.

> The normal setup would cost any CI/CD engineer a couple of days to setup and normally not more than an hour or 2 to debug infrastructure related issues.

> Deploying on Kubernetes usually involves rendering templates, setting CI/CD runners, generating on the fly kubeconfigs for security and so much more that setting up CI/CD would take not less than a month to have something that is production grade ready.

I've spent time on both. "Normal" (to a VM) CI/CD flows have pathological corner cases dependent on the state of the VM. Flux or Argo is not hard to set up, and the benefit of YAML is reproducibility: pods consistently start from a clean state.

Much of the deploy work I've done with deployment scripts on VMs is re-inventing features that are part of k8s.

YAML generation is straight-forward from the need to have it be configurable. One you understand the need for it … IDK what the problem people have here is. "setting CI/CD runners"?

> The idea of deploying that many components together with different versions and different maintainers makes it impossible near to have an error free Kubernetes cluster, from network CNI to controllers to custom operators things tends to breakdown most of the https

… this is always going to be a problem…? On a VM, I've got nginx, whatever handles certs, logs, metrics, security team's spyware^Wprotection etc. … all from different vendors, at different versions.

"things tends to breakdown most of the https" ???

> It makes perfect sense with all the network encapsulation over the cloud network virtualization over the real network well things tend to get crazy.

???

> To be able to keep track of the versions of all components deployed on your cluster

(same complaint)

> And it should you are opting in for a setup intended for huge enterprises that have huge budgets and don’t mind having Kubernetes because it is actually benefiting them.

This is begging the question.

> Bigger attack surface

While the API server is certainly a non-zero attack surface, this blows it out of proportion. People run salt on VMs, … see it's security track record: k8s fairs a far bit better. I've witnessed many orgs not set up some for of SSO, and try to manage VM access via SSH keys, and old keys do not get cleaned from nodes.

The standard tools work for the API server: you can firewall it off, if it scares you?

I would caution a young startup against running k8s: if you want to do it, you either need to have someone familiar with it who can teach, or eng who are capable of learning. If you have the all-too-common eng who don't like learning and who aren't familiar with it, you are going to have a bad time. But it isn't hard to learn: k8s has great documentation. But I've witnessed a lot of frustration from people who haven't RTFM.

We even use K8S in local dev env...wth?