presspot

I enjoyed this thought piece on technical debt vs technical rsik: http://pl.atyp.us/2015-01-technical-risk.html

This is exactly right. Marathon or Kubernetes running on Mesos or Mesosphere's DCOS is functionally similar to Borg.

Mesosphere is arguably the largest contributor to Mesos, certainly on par with Twitter, especially when you consider all the surrounding ecosystem. The company also secured permission from the Apache foundation with the trademark when the company was founded. It's good for the open source ecosystem to have companies productize and support projects, particularly when they are plowing millions of dollars back into the open source.

The whole point of the Mesosphere DCOS product is to make Mesos consumable by mere mortals: https://www.youtube.com/watch?v=UgJMlHdZEx4

Here's Mesosphere's take on Docker Swarm, orchestrating at scale with Swarm + Mesos

http://mesosphere.com/2015/02/26/deploying-with-docker-swarm...

Related: https://news.ycombinator.com/item?id=9039197

Also related: http://mesosphere.com/2015/02/11/yarn-on-mesos-big-data/

A good friend of mine used to teach ESL at San Quentin. He described to me in great detail the "pruno" trade in the prison, including the elaborate exchange of services for scrap fruit and pilfered trashbags from the commissary as well as the even more elaborate system for distribution and payment of the finished product.

Is possible, though I'd probably run Spark and YARN native on Mesos on bare metal.

Jim Scott from MapR compares Apache Mesos vs. Hadoop YARN https://www.mapr.com/blog/apache-mesos-vs-hadoop-yarn-whiteb...

Stunning

Here is the Github repo https://github.com/mesosphere/mesos-dns

Kubernetes on Mesos makes sense to me. When you run Kubernetes on GCE, you have all of Google's infrastructure... but if you run it anywhere else, how do you get scale and HA without Mesos? (Let's be frank: when is anybody going to run 100% of their workloads on GCE). I think Kubernetes on Mesos fills the 90% gap of all Kubernetes apps running outside of Google and maybe even on Google for app portability.

I see Kubernetes as more of a programming model, not an operating system. It provides a way to express services and have them scheduled onto a datacenter. The rocket-science of how you schedule those workloads and optimize them in the same partition as other workloads is what you need a DCOS for.

In the Mesosphere world, Kubernetes is a "datacenter service" which is installed on your datacenter so that you can run Kubernetes workloads. You might also want to install DEIS to run DEIS-organized workloads. Or Spark, for Spark workloads... and so on -- all multitenant in the cluster. This is what the DCOS is uniquely good at, and why it qualifies as a true operating system.

So, yes, Mesos predates Kubernetes by many years.

A bit of the history:

The Mesosphere DCOS is built around the Apache Mesos kernel

The Mesos kernel was developed at UC Berkeley in 2009 [1].

Spark was written as a sample app on top of it [2].

Ben Hindman and his colleagues at the UC Berkeley AmpLab had always envisioned Mesos as a kernel inside of a full-blown operating system [3]. They finally brought it to market.

[1] https://www.usenix.org/legacy/event/nsdi11/tech/full_papers/...

[2] "We have implemented Mesos in 10,000 lines of C++. The system scales to 50,000 (emulated) nodes and uses ZooKeeper for fault tolerance. To evaluate Mesos, we have ported three cluster computing systems to run over it: Hadoop, MPI, and the Torque batch scheduler. To validate our hypothesis that specialized frameworks provide value over general ones, we have also built a new framework on top of Mesos called Spark, optimized for iterative jobs where a dataset is reused in many parallel operations, and shown that Spark can outperform Hadoop by 10x in iterative machine learning workloads." ibid.

[3] http://people.csail.mit.edu/matei/papers/2011/hotcloud_datac...

MPI will run on top of the DCOS. That is the point. It supports a multitude of scheduling models all running multi-tenant.

The DCOS supports network virtualization. Changes in underlying infrastructure can be manifested all the way up the stack to applications and their schedulers. E.g., the DCOS can rewrite network routing tables based on placement decisions.

That is exactly right.

Is a 40,000 core disaggregated rack really that different from a 4 core laptop? Google pioneered this way of thinking [1]. tl;dr the datacenter is a computer and that computer will inevitably have an OS.

[1] http://www.cs.berkeley.edu/~rxin/db-papers/WarehouseScaleCom...

A mid-market SaaS company might have a $1mm/mo Amazon bill. I think cutting in that half would be meaningful to just about anybody.

Rip and replace of a hard drive is not hard when you have a datacenter that is completely self-healing. An intern on roller skates can do it.

the 1 admin to 600 machines quoted as the high end for traditional IT datacenter is, in my experience, a murky number. It's usually a ratio of people/virtual machines, not physical machines.

When you remove the VM smokescreen and count physical boxes it's more like 1 person/100 machines, which is abysmal. I've seen order-of-magnitude people efficiency increases with automation like we're discussing here.

>>> I don't know why many people think that they need to be at datacenter scale computing to benefit from abstractions like Mesos, it's completely wrong imo

My next startup will be built on Mesosphere. Faster time to MVP and no "go dark for 18 months" when I have to scale.

There are a lot of components to an operating system. It's not just the technology components, it's the product components and the business components. E.g., Does it have an API? Does it have an SDK? Does it have a user interface? Does it have an init system, a chron, a storage system, service discovery? Does it have an ecosystem of third party developers? I posit that the OS Checklist is fairly long and that no of the other systems you mention have the complete OS package.

It's very similar to mainframe computing, only now it's available to every business and, frankly, every business needs it to be competitive.

Mesosphere's stack is in full production at major companies, including one of the largest financial services companies and one of the largest consumer electronics companies. General availability is next year, but paying customers are using it in production today--at very large scale.

Mesos runs quite nicely on CoreOS. Mesos doesn't replace the native Linux on each of the boxes in the datacenter. The Linux on each box provides the execution environment.

Here is a short tutorial for standing up Mesos on a single CoreOS instance: https://mesosphere.com/docs/tutorials/mesosphere-on-a-single...

Datacenter and back-end apps simply don't fit on a single machine anymore. Every app of reasonable scale is probably a distributed system of some sort. That, and there are a new class of "apps" (or, more precisely, datacenter services) that were built to operate across fleets of machines from day one, such as Spark, Hadoop, Cassandra, Kafka, Elasticsearch, and so on

One thing worth noting is that the indirection Ben describes is, basically, the indirection that is already in the Apache Mesos kernel. In real world environments (e.g., Twitter), there is negligible overhead.

Also, locality and latency can both be expressed in terms of placement rules and schedulers on Mesos can use those rules to guarantee or express preference for task placement that optimizes around reduced latencies.

The point that I take away is that the ability to express your needs in a declarative way (e.g., "place these two tasks such that they have such-and-such latency) is much more scalable, flexible and resilient than coding to machine-specific internals. The latter is easier to update and supports delegation of responsibilities.

John Wilkes of Google puts it this way:

"Our own experience has been that allowing our developers unfettered access to the internals of infrastructure systems has been a problem, and we're moving away from that model as fast as we can.

Constructing large-scale complex systems with many interdependencies leads to brittle, fragile systems if they rely on internal implementation mechanisms.

Allowing internal customers to rely on internal implementation mechanisms has made it hard to adopt new technologies, because we only know what knobs they set - not why.

The fix for both is similar: describe the desired end state, not how to get there."