Hacker News Reader

Comment by kmac_ | Hacker News Reader

I've moved to a company that doesn't use Kubernetes at the moment (and that's a 100% calculated and rational decision). What I see, is that a lot of effort is put to provide functionalities that Kubernetes brings. In case of running a bunch of services, when you wish to do that in a stable and secure way, Kubernetes cuts down running costs. It covers so much cross cutting concerns that reimplementation of those capabilities is not possible unless you have heavy $$$ to spend.

View on HN

I think you're right to point out how much ground k8s covers and to replace every vertical that it integrates could be challenging/costly. But k8s is not a zero-cost abstraction, so I think the calculus here is often more nuanced.

In the case of my org, we optimized for the features we thought were valuable and amortized that effort over time. Notably this was early in k8s history (2014/2015), but the fruits of those efforts have aged well so far (8 years or so). Small code footprint to cover service discovery, cert provisioning, release orchestration, and configuration management. The whole devops stack is less than 3k SLOC. Service ecosystem is ~150 distributed systems, roughly about 5 million SLOC, running on just over 1k servers on AWS.

I think if the aim is not to completely replace what k8s does, but to cherry pick the features that give you some pareto distribution of value, sometimes its worth it to build in-house. Nothing wrong of course with going with k8s for many orgs, but in our case we didn't have to reinvent the whole wheel to live without it.

3k SLOC of devops code to cover a system of that scale is super impressive. And I agree, there's no reason to invest in k8s when only a small fraction of its capabilities is necessary (or we have a team that's already experienced). Otherwise we may end up bending our requirements to k8s abstractions (even though they are well designed).

At least for most hosted solutions, Kubernetes seems to be "cheap" (compared to other offerings at that provider) after you pass some reasonable threshold: something like 6-8 services each running 3-4 instances or so. This threshold seems to roughly end up being $500/m.

When I hear numbers like this I wonder what percentage of the compute and memory resources of that 18-32 node cluster — not to mention the engineering that went into making it work — goes into the “hard” problems of horizontal scaling, cramming stateful services into an architecture designed for stateless ones, etc.

You can actually get a couple of pretty beefy bare metal boxes for that budget. Or a couple of more modest ones for app servers plus a nice big RDS instance with all the trimmings. Based on past experience, that’ll get you to a few hundred rps for even a fairly complicated, poorly-tuned Rails or PHP app; your well-factored Go API server should handle 10x that pretty easily.

You might have to write some Bash or systemd unit files instead of a bunch of YAML, which may or may not bug you. I find shell easier to understand and debug than YAML-based scripting but YMMV.

Right, I don't think you can beat buying the machines, of course.

You don't have to buy - Hetzner, OVH, etc will happily rent you these machines dirt-cheap and that includes hardware maintenance & replacement.

Sure, that sounds some problems. Try to get your average CTO on board, though!

OVH may be problematic if you don't speak French (their English support sucks), but Hetzner is pretty famous and well regarded. Their network is great. They provide a lot of automation. The provide little tooling compared to AWS, but what is there works and it works great. Its an engineering-minded provider. Also, its VM pricing is the lowest. What's best is their egress pricing cannot be matched by anyone else in the US or Europe.

contabo is interesting too…

First time i hear contabo.

Same, we are on ecs and there is a lot of reinventing the wheel

Genuinely curious what things k8s solves that you are reinventing. I run ECS and find that using microservices and their managed offerings (ie RDS, SQS) we don't any complicated topography to do complex work.

1. Creating confined development environments containing multiple services. With k8s we can spin up a cluster locally, install all the dependencies and develop on it.

2. Remote development. With k8s we can develop right out of the cluster, ECS has no comparative.

3. Installing OSS software. K8s has loads of supported packages for OSS tooling.

Reply on news.ycombinator.com