Hacker News Reader

Comment by Thaxll | Hacker News Reader

There is no alternative as far as I know. Which open source or private solution scale above 10k nodes and 100k apps ( pods )?

View on HN

There are many private, proprietary systems that exceed those scales. They are usually bespoke to the applications that they run. I work with two such systems and know of others in finance, energy and scientific computing. Not to mention Borg at Google.

In the commercial proprietary world, clustered mainframes and supercomputers have addressed this niche for decades.

In the open source world, HashiCorp Nomad is the most analogous alternative to Kubernetes on commodity hardware, while SLURM is very successful for supercomputing.

I've also scale tested k8s to 15k nodes (in a limited configuration for a single application). At that point we ran out of underlying hardware budgeted for the test.

> There are many private, proprietary systems that exceed those scales

Yeah, and then you have to run (and pay for), some private, proprietary system. K8s might be a bit worse for this, but it’s at least “commodity tooling” and lots of common tooling supports it out of the box.

Usually the private proprietary systems are used because the application is at least one order of magnitude larger in scale than k8s can support.

They also sometimes predate k8s and migration is expensive.

HPC does a fraction of what k8s does though? Can you run 100k never ending jobs listening to networking?

From what I know HPC is basically run that short lived compute model on a very large cluster that's it.

> Borg at Google

Which is what Kubernetes is based on.

Kubernetes is based on Omega, which was an attempt to replace Borg using a more distributed architecture. Borg uses a single master with elections, with the replicas serving as hot spares.

Source: I was on Borg-SRE at Google for ~six years, then worked with Kubernetes at Stripe for ~five.

Yup, and while ISTR that Omega did (eventually) get rolled out, no one ever called it anything but Borg so you’re the best kind of correct :)

One of Nomad's major pitches is that it can scale larger than K8s. It's all over any comparison between the two.

And HashiCorp has a lot of success stories and experience making it work well. They can help you set up a continent-spanning cluster with 100k+ nodes.

HPC schedulers, most likely.

In HPC we see Slurm pretty often.

Can you use SLURM to schedule services? IE, long-running not-batch-jobs that expose network ports (often multiple)? That's what k8s (and borg) really are- service schedules, not batch schedulers.

You can schedule jobs without a wall clock time if the admin didn’t disable it, but it’s awkward. That will make scheduling MPI jobs hard/impossible.

Run your batch jobs in slurm and your deployments in gke.

I can run batch jobs in GKE. It looks like SLURM doesn't have a way to handle endpoints (IE, a port manager) so I agree, deployments will have to stay in something like k8s.

Mesos definitely does

did*. The community is dead.

Ah that's sad to hear, although I guess I contributed to that death by leaving

> private solution

whatever Google and Facebook use internally.

Reply on news.ycombinator.com