We made a very strategic decision to rebuild it. It embodies many lessons from Borg and Omega (both things we got right and things we botched). It is implemented in an easier-to-approach language (Go) which has an active OSS community. It specifically focuses on "legacy apps" (everything written up to and including today) and open standards (HTTP, JSON, REST).
I've never been shy about my opinion that I do hope to supplant Borg one day, but that day is necessarily years away. Of course no major project has ever used it, the whole thing didn't exist just 2 years ago.
Rubbish, and saying so undermines the rest of your points.
> Rubbish, and saying so undermines the rest of your points.
Hardly rubbish. Although there are open-source projects which use C++, I and many others avoid them like the plague.
I think he meant 'approximately ZERO' in the Spolsky sense, which is 'sure, there are some, but in the grand scheme of things they're indistinguishable from ε.
The Go community is vibrant and growing. Go is an easy language to learn (and I say that as someone who LIKES the power of C++) and it is not a total joke to ask people who report bugs to jump in and try to fix them. C++ is simply NOT approachable by mere mortals, and would have made for a very different community and a much slower pace.
And I say that as someone who detests many facets of Go - but it's just better at Getting Things Done than C++.
Much of the organic growth, as you put it, can also be described another way: as an accumulation of useful features and optimisations.
The language issue was addressed by another response. To claim there's no open source C++ community which is why it's written in Go is just bizarre. There's absolutely a thriving open source C++ community, but if having the biggest open source community was the driving factor in picking the implementation language then I guess you should have picked Java.
Truth is, a LOT of people don't write code. They write content and use pre-built code (think WordPress). Borg simply can not accomodate that very well. It's simple as long as you control things from soup to nuts.
Yes, some "organic growth" was useful features. But a lot of it was useless features, or features that are now obsolete but can't be removed because someone somewhere is using them, and probably doesn't have enough time to re-test without the feature (true story).
I don't think Borg imposes all that many requirements on jobs, really, and the few it does can be disabled. Or at least could.
But I guess we're probably wandering out of the area covered by the papers now.
The first and best is to make all your MySQL IO go to an external cluster filesystem or other remote IO system. Because MySQL supports pluggable storage, you could write an Hadoop FS storage manager. This has the advantage that if a single MySQL instance is blown away, all the committed data is available for a new replica to start reading. I don't know if Docker or other container systems support automagically turning local IO calls into remote IO calls (or whether that really makes sense in a MySQL environment), but that's a similar approach. Condor supported this through their remote libc interface.
The second is to use some sort of per-task persistent local storage. In Docker world, this would be a mounted volume- the docker host would manage the storage, and new containers would remount that storage. You could have a process that restored the local storage from a backup, and the use replication from a master to catch up.
The third would be to have some sort of per-container persistent storage (the Borg paper calls this an "alloc").
For the server, most people wouldn't have Apache bind port 80 inside the container- you'd bind another port, and use some sort of other mechanism such as load balancing to expose the web server on a standard port
I also don't blame people for being confused about Google's container infrastructure. Google has issued blog posts in the past that were misleading (at best) about the relationship between Omega, Borg, and Kubernetes.
A decade or history lead to the knowledge that a particular style of monitoring was needed. That knowledge lead to cAdvisor. Is it perfect, of course not, but it fills a need and is directly derived from Google's experience. I fail to see how that misattribution, personally.
Tim knows more about this than almost anyone, but I will add one point - we have used Kubernetes for significant internal projects, and plan to continue expanding its usage over time.
To Tim's point, though, it'll take time. The thought that you could move literally millions of lines of code and applications over to a new platform in just 10 months (the amount of time that Kubernetes has been GA) is... optimistic.
This is NOT in competition with Borg, though. Not yet.
So what's the real reason nobody gets to see this code?
Thank you for the details!