Hacker News Reader

Comment by jamesblonde | Hacker News Reader

I tend to agree. It's another case of - here is an API to access our products so you can play around with it. The scalable, powerful scheduler, however is not included. There are no other batteries that fit it, apart from GCE. Kind of like Blaze and TensorFlow - nice and shiny, but hollow (missing the good distributed filling).

View on HN

FWIW, you can plug mesos in as a Kubernetes scheduler. From the OSS world, that is about as heavy duty as you're going to get and proven on quite large (although not google large) 10k+ node clusters.

The internal google scheduling system is simply not appropriate for anyone other than Google to use. A huge number of machines, all alike, with a huge number of trusted binaries that can be multiplexed onto these machines without fear that they're going to break the jail and cause havoc (since there is a solid trail from source to running artifact). It's just not the reality that other companies exist in.

Scheduler is actually one relatively simple piece of the whole picture. If scheduling is a pain, then Kubernetes would just addressed. The fact that Kubernetes did not choose scheduler, means that it is actually not a big problem, at least not the biggest one.

Borg jobs are not trusted! The system sandboxes them, prevents them spying on other jobs data files, and assumes they might abuse system resources in arbitrary ways. The days when all Google machines trusted all Google employees is long in the past.

That is not an argument against releasing the code. Why would Google assign itself as gatekeeper? I personally could use this code on supercomputers now. I don't work for a supercomputing company; my use case is academic work and computational science. I absolutely have thousands of huge machines that I multiplex trusted binaries to -- and scheduling is not a trivial problem.

So what's the real reason nobody gets to see this code?

The code is literally millions of LoC, all of which needs to be audited for stuff we can't release for whatever reasons. All that code is built upon layer after layer of Google internal stuff. Open-sourcing Borg means open-sourcing Chubby, internal form of gRPC (older), hundreds of libs, monitoring infrastructure, etc. Net result is O(50M) LoC. And when someone sends us a patch - then what? The cost of doing it is simply prohibitive. I'd love to do it, it's just not practical and has no RoI.

That's a much more sensible reason! If the code is truly Google specific, then I agre. It sounded to me like the code was not released because nobody else has a lot of computers, which I found odd.

Thank you for the details!

As for "no other batteries that fit it" - I am confused. We do run on AWS, OpenStack, and other cloud providers, as well as on bare metal. It's not like nobody is using this thing. In fact, just a finger in the wind, I'd guess the number of people using it outside of Google Cloud is several times more than people using it on Google Cloud.

"Powerful scheduling" is such a tiny piece of what Kubernetes does, it's funny. Yeah, Borg's scheduler is faster and more scalable and has more features. It also has 12 years of optimizations under its belt. I have 100% confidence that, should Kubernetes be around 10 years from now,this will be a non-issue.

Sure, even though having multiple schedulers support (for different type of workloads) would be great to increase the cluster utilization, which is one selling point of such systems. I understand that Kubernetes is developed in the open and with the community but the heavy marketing as the "solution you can use now, directly from the creators of Omega" makes some people think it's ready, perfect and will fix all of their problems, but that's simply not true.

We have multi-scheduler support in v1.2 :)

Great, the docs are a bit bad for that, I'll have a look at the code :-)

Reply on news.ycombinator.com