Peloton: Uber’s Unified Resource Scheduler for Diverse Cluster Workloads | Hacker News Reader

Peloton: Uber’s Unified Resource Scheduler for Diverse Cluster Workloads

eng.uber.com

54 pointsr4um16 comments

>Kubernetes hasn’t been able to scale to the large clusters that Uber requires, i.e. 10,000-plus, nor does it support elastic resource sharing. It is also not the ideal scheduler for batch workloads due to the high-churn nature of batch jobs.

Feels like a semi-weird justification for NIH. Isn't there pretty active work in addressing most/all of those objections in k8s?

Yes, though the problems are real.

In its current form Kubernetes can only handle a cluster size of 5,000 nodes: https://kubernetes.io/docs/setup/cluster-large/ and high-churn batch workloads don't work well.

Kubernetes does support elastic resource sharing (assuming I understand what they need) via horizontal pod scaling and the cluster autoscaler: https://github.com/kubernetes/autoscaler/tree/master/cluster.... Although there are rough edges here, I was surprised by the quality of the code in the cluster autoscaler. I kept thinking of new edge cases and expected it to fall over, only to find out that they'd thought of that problem and it handled it gracefully.

For batch workloads something like Argo can help: https://github.com/argoproj/argo.

So basically I agree, it probably would've made more sense to build something on top of Kubernetes. But Uber's a big company with a lot of engineers so I guess they have the resources to roll their own solution.

Didn't they also have a weird "Postgres can't suit our needs so we're switching to MySQL" argument?

yurp

It's interesting how big of a difference it makes when you allow local disk I/O, and you need to schedule around it. At Google they disintermediate storage and all (not really all) I/O goes to their cluster FS (Colossus, sort of). They don't have to schedule around I/O resources because every process has access to the full I/O resources of the entire cluster at any time from any node. By contrast as soon as you let some open source or off-the-shelf commercial thing leak into your operations, it will demand ordinary POSIX disk I/O and then you've got big problems. I propose that some companies would actually be better off concentrating on disintermediated storage more, and I/O workload scheduling less.

>I propose that some companies would actually be better off concentrating on disintermediated storage more, and I/O workload scheduling less.

I feel like this isn't possible outside of Google. I've never had good performance (well any performance that you would feel comfortable running a database on) on any storage medium that wasn't POSIX disk I/O. I may be ignorant, but I've never seen a good disintermediate storage solution that is performant on something like AWS.

> all (not really all) I/O goes to their cluster FS (Colossus, sort of).

Well - sorta. Cluster FS just means disk I/O is now network I/O (EBS is a bit like this). Now you push the hairy problem to a different area: your network topology starts to matter a lot. You might end up having to think about not scheduling too many I/O heavy jobs behind the same switch or something. AND disk I/O is now competing with network I/O. AND it might now be hard to predict the destination of where all that I/O is going to be when scheduling the job, whereas previously it used to just be local.

So the problem doesn't go away. But maybe you say it's just easier to wire everything up with gigE / infiniband so that you're massively under-capacity, and call it a day, and that might be a fair tradeoff.

The “Storage as a Service” needs to run somewhere. You can’t escape from running a storage aware scheduler. Either it will be the same as your primary scheduler, or storage services will be special snowflakes.

There are always daemons on the machines that aren’t under the control of the cluster scheduler (init or its equivalent, etc). The thing that exports the storage assets of the box can simply be one of those things.

I'm always bummed when I don't see Nomad listed in blog posts like these. It has quite a few powerful features that people overlook.

CMU has a "self driving" db with the same name https://github.com/cmu-db/peloton

Also, there's a B2C company called Peloton [1].

[1] https://www.onepeloton.com

Nothing in the article about this being open source or available to the public?

why did you think it would be?

from the article -- Peloton is built on top of Mesos

They use mesos, and the article explains why they added something to it.

Reply on news.ycombinator.com