Hacker News Reader

Comment by mikepurvis | Hacker News Reader

I'm only a light k8s user, but isn't part of the pitch that it actually gets you out of having to run parts of your infrastructure? Like, there's no point spinning up EC2 nodes just to run microk8s or k0s on them— have ansible scripts or whatever that just run your application directly.

But the calculus changes if instead of EC2 nodes you can have a helm chart that deploys directly to a managed kubernetes services like EKS or GKE?

View on HN

Eh, I've found kubernetes more complicated than the interface of a cloud provider, you always have to do the cloud provider as well for some things, and the benefits of the pre-baked helm charts or whatever else just aren't there until the relatively small difference in efficiency in each way of doing things makes a real difference as a result of scale.

If you have a lot of experience with any interface, you can do it really efficiently, but if you don't, kubernetes can make your experience worse than anything you'd come up with on a cloud provider.

Fun with k8s starts when you realize it has no LTS and gets updated on a fixed schedule. Any given version is only officialy supported for about 1 year and a half.

Then you'll need non trivial amount of knowledge of how it works to avoid many pitfalls (the helm chart will probably only go so far, it will still need small tweaks for your specific use case)

So it's a combination of upfront time investment, ongoing tuning, and recurring maintenance events.

Honestly, I find Helm to be a trap that is going to reduce your capability to run service reliably - or understand it. It's probably one of the most annoying bits in kubernetes ecosystem

> isn't part of the pitch that it actually gets you out of having to run parts of your infrastructure?

That's a big part of many k8s pitches I've seen but in my experience it hasn't been representative of reality (or at least for every part of infrastructure you no longer need to run there's a new part of k8s architecture/config to learn & manage).

For me the selling points are:

1. For the time-cost of upfront complexity and setup, you get extremely easy and quick subsequent addition of infra/tooling & scaling thereof. 2. There are certain patterns of tooling that k8s enables (relatively easily), that would have been difficult/impossible with more traditional setups.

I don't think it makes infra management overhead less, it just reorganises it so it's more upfront and then less later while scaling.

Can't you run K8 on Fargate so you don't have to manage EC2 nodes themselves?

Even with ECS which is way simpler than K8 for small teams using Fargate has been incredibly beneficial in terms of debt/time.

Then next step would be to move the entire application onto Lambda so we're not paying for downtime, but that's a whole headache and a half.

That's still a bunch more complicated than other hosted solutions. I once worked for a 4 year old company whose entire infrastructure consisted of a managed postgres instance + a moderately sized single VM (and a duplicated setup as a staging environment).

What are the benefits of k8s over docker-compose for small/medium business?

All of the cloud providers offer k8s as a service, for one.

I should specify I was thinking about VPS. Coming from AWS to Linode VPS with Caddy + Docker felt great, Im wondering at what point (N requests per second ?) it becomes tech dept

Yeah that's definitely something that needs to be specified.

> Coming from AWS to Linode VPS with Caddy + Docker felt great

I guess that depends what you were using with AWS, but Fargate + k8s is a pretty amazing workflow.

> Im wondering at what point (N requests per second ?) it becomes tech dept

I make these calls for our org, and I wouldn't call it tech debt based on the performance or scale, but because of maintenance. As an example, if you needed to hire a new person/replacement person for your team, would it be easier to find someone who can maintain k8s or docker compose?

The other area it shines for is the ecosystem. If you need to add monitoring to your app (e.g. with Prometheus), adding it to a k8s cluster is `helm install prometheus`

thanks for detailed response! how big is your org? My concern is that there is a cost to magic of adding deps like that. Complexity goes up and it is harder to understand what went wrong and where are the performance bottlenecks. I guess my question is at what org scale the tradeoffs are right

> how big is your org?

My last team was the same, there was ~100 developers on the main app, ~50 on the ancillary services, and a ballpark of ~10 on the "platform". I was just a team member on that team. Current org is 15 devs, 30 people total.

> My concern is that there is a cost to magic of adding deps like that.

Of course there is, there's such thing as a free lunch. It's up to you to decide which is right for your team/org. The cost of doing it manually is developer time, tech debt and the bus factor. The cost of using helm is introducing an abstraction. Profesionally, the bus factor and developer time of home rolling a solution make relying on existing solutions a very appealing proposition. Having to maintain a set of half baked shell scripts is exactly what tools exist to solve, and I've had more issues with deployments where someone has bunged together a bash script that makes varios assumptions about tooling, versions, installation paths, working directories, path formats, etc. than I have with problems related to our actual deployment infrastructure.

> I guess my question is at what org scale the tradeoffs are right

The cost of switching is huge, and likely not worth it unless you have genuine problems that would be solved by moving. For a greenfield project, frankly I'd think you're wrong to use docker-compose on any new project. Either you're managing the underlying infrastructure (ec2/vps style) or you're not (fargate/DO App Platform style), and in either case docker compose requires roughly the same amount of investment for a smaller return on a less actively developed product objectively solves less problems and requires more work to integrate.

[0] https://github.com/marketplace/actions/deploy-to-kubernetes-...

thanks for your insights! I had an impression that k8s is more for enterprise stuff, I will definatly give it a try!

While the specifications are more complex, I find much more sensible in management than docker-compose, and allowing easier expansion (not scaling the service, expansion of underlying hardware) which means keeping costs lower can be easier.

Could you specify what you mean by "management"

One of the systems I set up recently I decided to go the docker-compose way. Quite quickly I found out that either the tool doesn't support (or at least isn't exactly interested in supporting) multiple compose files, or I am too stupid to find out the right combo of options, which resulted in a growing yaml file as the setup of dockerized components rose on that specific server.

Whatever claims about user friendliness, I didn't figure a stable way to do deploy separate docker-compose.yml's from same user at the same time.

Not using Compose in production, but our development setup is split across multiple files - you can include them by passing -f flag to Docker Compose (I wrote a small wrapper script that does that automatically)

Reply on news.ycombinator.com