Hacker News Reader

I'm an engineer at one of those companies you mentioned who worked directly on the Mesos+DC/OS platform.

Mesosphere's support was pretty much nonexistent as far as we were concerned. We had major issues open for years without any significant action. They were never resolved. We had to solve most problems ourselves.

We got so fed up that a few of us worked late nights (often up to 2AM) on a bottom-up skunkworks project to replace Mesos with Kubernetes. That project was exceptionally successful. (I'd like to point out that we are not in the Bay area and are adults with families and small children- we hated Mesos so much we were willing to stay up anyway.)

In short, Mesos has a lot of great theory but we hated it in practice. Kubernetes has some theoretical flaws but it is better experience for practitioners.

100% Agreed. Ex-Mesosphere employee (I joined in 2015 and left/was fired a year later)

The dominant ethos at Mesosphere was that they already won, and were poised to become the next 'cloud orchestration' above the cloud services. But the managers also had no empathy for developer experience -- the majority opinion was "distributed systems are hard, developers don't deserve to have a good experience", despite the new cool easy-to-use distributed systems cropping up every week. The company even developed a sham 'community edition' that was designed to fail. One person complained on our community slack channel that he left his cluster running overnight and was charged $600 for the 12 hours of use.

Within a few months of joining, I started to point out their odd technical decisions (e.g. why they decided to build their enterprise edition on Core OS rather than a trustworthy distro), and was eventually chopped for speaking up. I was fired right before a massive company offsite. When the rest of my team came back, they along with my manager were fired, too. The messsage was: we didn't need your whole team but because you spoke up we had to double screw you.

Ex-DC/OS user. This explains a lot.

One thing I'd like to say is that CoreOS was the best component of DC/OS by a long mile. We still use its fork (Flatcar) today.

That’s cool! The issue is that when you’re selling to a massive legacy institution, the infosec on a new Linux distro that self updates to new patches is really no bueno. We lost many months on this piece and on key deals. I heard from the grapevine that they ended up having to port DCOS to Ubuntu after all, soon after I left.

Yeah we disabled the auto updates and orchestrated the updates ourselves. Saved our bacon multiple times when kernel/systemd/docker bugs got pushed out.

> Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it

Brilliant on paper.

The idea was influenced by MR workloads on Borg. Because MR is so large a workload pattern on Borg, that it benefits from having its own scheduler, and Borg also benefit from unnecessary meddling.

The idea was only brilliant for a very small number of human users, both inside Google and in the broader industry. In terms of CPU & memory usage, mesos' approach should always take great share, because workloads that in the category of needing their own scheduler always is the big ones.

But as an open source product, its adopters are far off from that group. So you can see that all early adopters of Mesos are large corps. I seldom hear any startup applaud Mesos.

Precisely, I have repeated many times when Mesos was still relevant, that its model makes it costly to get started. It will never succeed unless it start to package a product that can provide the scheduler part.

We all know what happened after K8s. K8s works because Borg already worked for 12+ years.

> an elegant implemention that allowed for much more than what Kubernetes was desgined to.

By design K8s is more capable then Mesos. Because K8s includes scheduler. And numerous other capability. Heck, excluding performance and scalability, K8s is a level a bove Borg in terms of feature and capability. Mesos is far behind Borg even, not mentioning K8s.

Mesos does offer better scalability, on paper. I had no experience though.

> Kubernetes was supposed to be scheduler for Mesos and Google invested in it.

I never heard of such plan.

>> Kubernetes was supposed to be scheduler for Mesos and Google invested in it.

> I never heard of such plan.

Oh, there was a plan:

https://github.com/kubernetes-retired/kube-mesos-framework

But it wasn't really in anyone's business model interests, even if it may have made sense for customers and the community.

Sure, this type of early idea happens to any high-profile project.

I meant to say that that I never heard any significant decision maker had that plan on K8s' future.

As an early participant, while someone might have had this plan, I certainly did not hear this ever stated by anyone. Having the ability to run on Mesos was discussed as interesting and in the spirit of community but the work we did treated Mesos as a source of learning but not as a serious thing to integrate. I’m sure others considered it, but as justicezyx says it was an inclusive idea but at best something a few people cared about.

This is much more the real story but I doubt most folks will ever hear it.

Totally agree. Mesos project always had a trouble with governance and didn't build out the larger community the way other projects did. If they had built that coalition, made the project more accessible to others, who knows, maybe it would have gone differently.

Then again, k8s succeeded in part due to the Google reputation (even if undeserved) and it's use of GoLang. I always found Mesos' use of C++ meant many of the users (often developing in Java, Go, or Node) just wouldn't contribute.

I think the problem went deeper than that actually. Mesos required that applications use a distributed system API to interface with the executors. This meant it was very challenging to write a distributed system upon Mesos, even though that was ostensibly the very thing that Mesos was meant to solve. When K8s came along and made everything manifest and YAML-based, we sort of knew that we were screwed because of how much simpler that was.

Eh, maybe. I see your point.

The thing is, Mesos on it's own isn't k8s. Mesos is a scheduler. k8s is a full stack. So it's always been a bit off to fully compare the two.

If your system fit clearly within what k8s did well, then you were fine and the stack worked.

If you had a more complicated architecture with parts that didn't play well with containers or k8s' scheduling model, life became pretty difficult.

That's one reason I liked Mesos is you could build the stack you needed for _your_ infrastructure.

Granted, I don't think most shops had the kind of problems that warranted Mesos, let alone k8s (that's still true today). But k8s is "good enough" for lots of problems and understandably that's where the community went.

> Everything shifted towards DCOS and everyone was pushed towards commercial licenses.

This was then partially diluted when Microsoft decided to deploy DC/OS and, from what I've heard, pushed Mesosphere to open source large chunks of DC/OS.

And then ACS was pretty much abandoned not long after that.