Hacker News Reader

Comment by grumpy-de-sre | Hacker News Reader

At this point I'm pretty much convinced these repeated "Kubernetes is bad for startups" rants are some kind of FUD campaign (probably grass roots from people who've missed the containerisation and declarative infra train). Kubernetes is actually pretty great for the vast majority of use-cases. Sure if you don't have experience with kubernetes leave learning it until after you've hit product-market fit. But rejecting the advances made in this ecosystem is a red flag upon itself.

My ultimate MVP starter pack is probably a big fat VM running a single node K3s instance and a cloud provider managed database. Software wise nginx-ingress / cert-manager / argo-cd is a pretty good default set.

View on HN

> Kubernetes is actually pretty great for the vast majority of use-cases.

From my brief glances at it, it seems like a great solution for very complex use-cases. But "vast majority"? The cost-benefit seems way off unless you have a very different sense of "typical use-cases" to me.

Most people are building simple web sites and apps that run on a single server, surely?

Agreed. OC is overstating it. It’s not that the anti K8s crowd is necessarily anti-containerization, it’s that containerization doesn’t necessarily require Something as complex as K8s.

I’m invested in Cycle.io, which is a company that simplifies containerization and infrastructure (major oversimplification of the company). But a lot of customers are refugees from K8s, whose appetite exceeded common sense. Their infrastructure ops teams are costing them 6-7 figs that they can shed with something like Cycle, because K8s is simply really complex.

This is going to become really really important in the next 3-4 lean years we have coming our way. If you’re a K8s user, and spending money you can’t afford on an ops team to manage, you owe it to your company, investors, and customers to explore options to reduce this type of burn.

P.S. Intentionally not making a hyperlink for Cycle.io because I’m honestly not trying to shill here.

> Most people are building simple web sites and apps that run on a single server, surely?

I believe so, which makes a comparison between k3s and docker-compose very interesting.

I learned the basics of docker-compose in about an hour and I don't see myself outgrowing it in the near future.

> Kubernetes is actually pretty great for the vast majority of use-cases

Its a pain in the arse, and you're on the hook for looking after it.

I'm captain infra, so when Managed DBs came along, I felt threatened. But then when I used it, I realised that, yes, they are expensive, but I don't have to fucking worry about them. One touch deploy, proper backups are an option away, as is proper role based auth. K8s is like having a DB admin. Everything revolves around them, and it can be limiting, unless you have a specific use case.

K8s is useful for a specific role, but for most people who've outgrown a single machine/bunch of machines, ECS is good enough. Yes, there are fancy things you can't natively do, but you probably shouldn't do those just yet anyway.

Where K8s comes in useful is probably limited to:

1) mixture of real steel and cloud

2) A cluster shared with a large number of team, each deploying services that rely on other people's stuff

I've been managing clusters in one for or another for many years. K8s is certainly better than swarm, but its really not that great as a general purpose "Datacentre OS".

in short, K8s is your generation's XML.

Kubernetes is the React of devops. You wont get fired for picking it. And there will be a tonne of support available online, as well as courses, and experienced people you can hire. Also there is cloud managed k8s which takes most of the pain out of it. And plenty of out the box stuff.

It is not a bad choice if you want to do the ops yourself for some reason. If you don't then use a BaaS or PaaS, that might be easier and probably not much more expensive, but will lock you in.

This has always been my observation/thought as well. It’s “ no one ever got fired for picking IBM” all over again. Cute advertisement slogan, but it’s based on a true/real mentality, and a dangerous one at that (in terms of reaching potential and managing burn).

You seem to be saying there's nothing inbetween K8s and "use a BaaS or PaaS"?

What about something that does the subset of what K8s does that is the simple bit and covers most use-cases outside of complex clusters?

I get the strong sense people are just misusing it for the wrong types of task. I've seen this happen countless times in tech. Development by CV.

Docker Swarm, Nomad, ECS?

I've seen K8S recommended for single server installs with a couple of services. i.e. people jumping on instead of vanilla Docker/Docker Compose.

Following that argument why use Docker? Just spin up a VM and install it on that.

It is like asking for something between vanilla JS and React. Or something between zipping your src folder and git.

The problem is, while this is in theory good, the weight of knowledge and support for React (and git) make it worth putting up with a bit more complexity. That support covers: Stackoverflow, Colleagues, New Hires, Cloud Support of these things (Vercel for example using both Github and React!)

You learn the tool, then you are set for 10+ years, probably, in both cases.

In the Kubernetes case, the killer thing is it very easy to set up a cluster on cloud platforms. I am not so sure about say Docker compose or the other ones. I am in the Azure world and Azure dropped support for direct container running, and now you need to use k8s if you want something cloud managed. (As far as I know).

"Containerization and declarative infra" are only a small fraction of what could be done with real network distributed OS's as early as the 1980s and 1990s, and with far lower levels of overall complexity. That "train" has left the station a long time ago, it's going full speed and there are no brakes on it either. Of course you can keep chugging along with clunky K8s and call that a "good default choice". But that's just being complacent about what real progress looks like.

So where is this real progress happening today?

I will say I think what the fly.io guys are doing with VMs on the edge and in-process distributed databases is pretty exciting.

Could you give some examples of these network distributed OS's that are going full steam ahead? Also interested in why you think those are less clunky than k8s. :)

Plan9 has basically invented the idea of namespaces, which are fundamental building blocks of containers. Plan9 also invented 9P file protocol which is used in WSL2.

Plan9 is the ultimate distributed OS. Running programs on different computers is as easy as mounting a remote /dev/cpu file to your process' namespace. Virtual desktop is as easy as mounting /dev/draw. And so on. Many such complicated things that require ad-hoc solutions on Linux are solved on a fundamental level in Plan9.

I wouldn't say that it's going full steam ahead, since its use in the modern world is fairly limited, but the ideas were there, and they have influenced the development of Linux.

Distributed OS hides distributed nature from the application. Kubernetes does not try to hide it. It allows developer to embrace it. Hiding network latency is leaky abstraction. Kubernetes does it right. It does not pretend that there's no network between your services. It's the other way around: there's always network between your services. Sometimes it's loopback network, sometimes it's real network, but you're aware of it.

I don't think that that engineers who built Kubernetes were looking at Plan9 and saying "ok guys, it's too easy to do things, we must make it harder so people are aware of the network".

I think the more believable scenario is "hey let's see what we can do with the existing Linux systems and sockets and stuff" and then importing the network into the model because it was the simplest solution.

Then again, whether or not should applications be network-transparent-by-default is a discussion for itself. I believe they should. Every program (that does not drive hardware directly) is just a piece of code that glues various OS APIs together. In the case of Plan9, the APIs are the filesystem, and the filesystem can be modified with 9P protocol. If we can use the same local program to perform an operation on a remote machine without any modification, why the hell shouldn't we.

This! Real innovation is doing something without introducing unnecessary complexity and enormous codebases.

In my latest project I skipped the managed database and started using a postgres operator. So far the dev experience and performance is good.

The PostgreSQL operators are definitely usable, pretty much none of their limitations are going to hit you on an MVP. I think a lot of the FUD about "don't run databases on kubernetes" came from a time before we had statefulsets and persistent volumes.

Which operator did you go with out of curiousity?

I'm using the Zalando one. It only took a couple of hours to get it running (first time running it) and integrated in graphana. Includes incremental backups to S3. It really feels like a cloud independent managed database.

> probably grass roots from people who've missed the containerisation and declarative infra train

I am curious. How much work would it take to get on this train in 2022? Sounds like more of a cult than a valuable technology proposition when presented in this manner.

could you elaborate on the "ideal" argocd workflow?

let's say devs push to gitlab, which starts a pipeline, tests run, container images get built, then in the last step in the gitops repo a version bump happens and then argocd will pick it up automatically?

(and if one wants to be able to rollback then that's the same workflow just instead of version "bump" the pipeline tags images with the commit hash and at the last step sets the version to the right tag?)

are there any gotchas, best practices? :)

thanks!

For deployments using Argo, yea that's pretty much it. You can set it to automatically deploy to your target environment as soon as it detects changes to your k8s mainfests, or you can require a manual "approve" step which is a push of a button for configs that are out of sync with what is applied to k8s.

For rollbacks, you have a button in argo that lists previous deployments and you can just choose which previous deployment you want to go back to. It works pretty well those (thankfully few) times i have had to rollback any deployments.

Can it run smoke and acceptance tests? Or does it require calling external services?

For automated testing the argo-workflows project is amazing and worth looking into.

I think when you look at the entire argo suite you start seeing something that could really disrupt the way we use products like gitlab, particularly for startups.

For smoke and acceptance tests you can use the sister project argo-rollouts: https://argoproj.github.io/argo-rollouts/

But that's going past an MVP/startup toolset.

Not the parent, but this is exactly what happens. ArgoCD is pointed to an "application" chart which just points to a path of a helm chart (of your app/sevice) in Git. So when your CI changes the image hash in the values file in your helm chart, ArgoCD with notice that, and change the deployment resource's image hash. It is just a nice system to have resources in sync between your git repo and what you have live in the cluster. Of course, if you change anything else, ArgoCD will change it too.

You're also right, rolling back just includes changing the hash back to whatever you had, or to a new hash which was a result of a git revert or whatever. Also the good thing is that if that newly deployed service is very broken (that it doesn't pass the k8s health check), ArgoCD will hold on to the old ReplicaSet and will not let your service die because of it.

What's also nice about ArgoCD is that you can play a bit with some service/application in a branch. Say you have some live service, and you want to adjust some configuration of it. Usually third party services have a lot of options to set. The problem is that you're not 100% sure how to get what you want, so doing a pull request for every small change can be very slow and exhausting. To work around that, you can point your ArgoCD's application chart to a chart which is in a branch, test/dev/fiddle with just pushing to that remote branch, and when you're satisfied, you merge your branch, and at the same time point your ArgoCD application manifest to point to the master/HEAD for that chart. In effect, at this step, only your Git repo will be updated, your service already has all the changes so ArgoCD will do nothing. That way you can iterate faster, and undo whatever regression you've introduced just by pointing ArgoCD to watch the master, not your branch (or you can just reset your branch to be indentical to master).

Reply on news.ycombinator.com