I invested in Kubernetes early and always meant to give the others a try (so I could at least know the differences), but never got a chance to.
I found it dead simple to get up and running with Nomad, but it is (perhaps intentionally) missing a lot of features of Kubernetes. For instance, if you want load balancing and auto-scaling, you need to rig it up yourself. In K8s, you set up a service and horizontal pod autoscaling and you're done.
Boku is Japanese for "I" (cf Watashi).
Beaucoup is French for "a lot".
The one thing DC/OS had absolutely nailed was the web interface and bootstrap. The DC/OS interface is brilliant as it allows you to explore all the possibilities and actually have a overview of what is happening.
Also, it was a lot easier to reason about because everything is contained in a single marathon job. No need to split everything up in deployments / services / ingresses. A single JSON file is all you need for DC/OS. Less to think about.
The downside of all this is that DC/OS feels like a solution for a theoretical problem, while kubernetes is the solution to practical problems.
What did you think of DC/OS as a base platform? From an ops perspective, I’m finding a lot to like at least conceptually. Especially the idea of managing one “kind” of cluster that then manages many additional kinds for you. But I have yet to actually use it.
Practically, I think kubernetes with helm charts gets will get you at least 90% of what DC/OS offers.
Also, running Kubernetes itself is complicated enough, running it on a different scheduler will just expose you to the pains of both schedulers.
The idea was always that you should manage this infrastructure just like they are managing containers.
discloser: I am the PM of Kubernetes on DC/OS.
Uber took advantage of this by running thousands of nodes of Cassandra in DC/OS.
The setup can get a bit complex quite early and won't be worth the effort to manage 8 nodes, when you begin to scale to around 20 nodes running a bunch of different workloads (batch jobs, web services, etc.), can avoid provisioning on the application side, etc., then k8s begins to shine more and pay back the investment.
Two clicks to get 1-1000 nodes on GKE. The work is to learn the yaml syntax and the way to deploy to GKE... but most apps need to learn something about how it will be deployed (be it how to use ansible to deploy vs how to setup on k8s vs how to use serverless). But you need to do this for 1-1000 nodes, may as well just do it once...
Kubernetes is more flexible and more powerful, but it's also a lot more complex. You either go with one of the managed distro's which take on some of that complexity but also reduce your flexibility, or you manage the whole thing yourself.
https://platform9.com/blog/kubernetes-docker-swarm-compared/
Edit: updated url to a more recent version of the article
Debugging problems can get really hairy because of the two layer split, and it's just not worth it if one is running Marathon as the only framework.
I came to the conclusion that Mesos' biggest strength is giving people the option to write their own framework. It's not difficult, and gives you a lot of power over execution.
Marathon itself appears to be very simple, but has some really weird shortcomings. For instance, it's not possible to submit a deployment and then check whether everything went well, except if you manually compare the deployment before and after (this was still true as of 1.4.x).
Furthermore, Mesosphere abandoned the Marathon GUI, and I got no clue how the work of transitioning that part into the hands of volunteers are coming along, but this was essentially what tipped the scales for us.
(We're currently migrating from Mesos to K8s.)
Disclaimer: I am the PM at Mesosphere.
Unfortunately nice features like "Resource quotas per node" are not available in the open source version, this made Nomad a no-go.
I don't know where nomad could compete to generate income, but it definitely wasn't there...
Kubernetes is built in a way that it's easy for people to write something that watches cluster metadata and perform actions (controller pattern), as such a lot of functionality is built in gradually overtime just like that. I'm not sure when resource management first came on the scene but it's been around for a while.
You can also use Kubernetes to manage resources completely unrelated to Kubernetes by bringing Custom Resource Definitions ("CRDs") into play -- that's when you create a "fake" Kubernetes resource that (ex. VirtualMachine) manages some resource that's actually on the machine. The combination of CRDs and controllers to manage them is called the "Operator pattern"[2] and it's gaining a lot of hype right now as people wrestle with it conceptually but it's been around the whole time.
[0]: https://kubernetes.io/docs/concepts/policy/resource-quotas/
[1]: https://kubernetes.io/docs/concepts/configuration/manage-com...
We like to create a new framework for each instance of the job (so we can track slow runs, trial-run new versions), so we can have anywhere between 0 and 50 similar frameworks all running and vying for 100% of the cluster.
Naturally, we have cluster-level scaling, so we add nodes until we hit our max spend per hour, and then jobs take a bit longer to complete.
Weirdly enough, Mesos resembles a system I was building in my head that I thought could compete with Kubernetes... Taking the resource supplier (agents in mesos-speak) and consumer (frameworks in mesos-speak) paradigm to the extreme.
[0]: https://mesos.apache.org/documentation/latest/architecture/
I worked with Mesos a bit last year and miss the sophistication of the scheduler. Granted, I never ran it at the same scale, but I felt like the approach was much easier to grok.
I've seen a few projects floating around meant help dynamically tune resource utilization in k8s. By this time next year, I imagine those will be fairly commonplace. I'm definitely looking forward to using descheduler:
I know that's a minor issue, but after getting used to it, it's really weird having to do manual refreshes.
Also, the Kubernetes UI feels too clutered, but that's probably just because I'm not used to it
Disclosure: I'm Director of Product for Weaveworks.
- works out of the box
- has much less overhead in getting something started
- tracks dependencies between services