Hacker News Reader

Comment by weavie | Hacker News Reader

So the article seems to suggest that for smaller clusters use Kubernetes, for larger ones use Mesos.

Would people agree?

Author here. I would say it's a bit more subtle than that.

If you have a very large cluster (1000s of machines), Mesos may well be the best fit, as you are likely to want Mesos's support for diverse workloads and the extra assurance of the comparative maturity of the project.

The big difference with Kubernetes is that it enforces a certain application style; you need to understand the various concepts (pods, labels, replication controllers) and build your applications to work with those in mind. I haven't seen any figures, but I would expect Kubernetes to scale well for the majority of projects.

The one thing I don't totally understand about the Kubernetes scaling issue is that you can ostensibly use Mesos as the scheduler for Kubernetes, which naively to me seems like it would allow Kubernetes to scale to the same size as Mesos, or is the DNS and Service stuff the part that doesn't scale as well?

I think Kubernetes on top of Mesos may become a reasonably common set-up, largely because it allows diverse workloads that can drive better usage and efficiency.

I'm not sure what the trade-offs are with running Kubernetes on Mesos - that would make for an interesting article.

(Mesosphere co-founder here)

Running workload-specific schedulers like Kubernetes on Mesos is one of its fundamental ideas. From the paper:

"It seems clear that new cluster computing frameworks will continue to emerge, and that no framework will be optimal for all applications. Therefore, organizations will want to run multiple frameworks in the same cluster, picking the best one for each application."

https://people.csail.mit.edu/matei/papers/2011/nsdi_mesos.pd...

Absolutely! We're actively working on this here at Mesosphere, in partnership with Google. You can check out the project at https://github.com/mesosphere/kubernetes-mesos

You might also be interested in a talk I gave at the Kubernetes launch in August about our work coming the two: https://youtu.be/aXcdHwQ5GgQ

Full disclosure, I work at Google on Kubernetes.

a1r is exactly right - Mesos does great in virtualizing your data center, Kubernetes is a framework on top of that.

There are a bunch of scale limits in Kubernetes itself that are being chipped away at; the bottleneck is not the scheduler, so plugging a new scheduler in won't make the system more performant.

Here's a blog discussing the current state of K8s scale: http://blog.kubernetes.io/2015/09/kubernetes-performance-mea...

Note that Bob Wise at Samsung has been driving some horizontal scale testing, and has got a 1000-node cluster up and running, so that's a current "best case" scale number.

Here's Bob's video of a 1000 machines running on CoreOS' Tectonic product: https://www.youtube.com/watch?v=1bHUsBhPL20

(work at CoreOS)

> The big difference with Kubernetes is that it enforces a certain application style

In our experience, it isn't so much that Kubernetes enforces a certain style as that it doesn't force services to understand the scheduling layer. A Twelve-Factor-style app will deploy very nicely and easily into a Kubernetes cluster.

If you're facing endpoints at the outside world and would like to do so easily and with pretty good reliability, Kubernetes on Mesos is probably the way to think about it.

Or should, when the K8S-Mesos stuff is a bit more mature.

Reply on news.ycombinator.com