Was involved in this research project for a few months in early 2019. Feel free to post some questions that are not sensitive to internal details, and I can answer them here.
Edit: not one of the coauthors, since I left the team in April 2019.
Edit: not one of the coauthors, since I left the team in April 2019.
* Borg is primarily for managing hardware resources. K8s, in comparison, is for managing applications. I often call them a complementing twins. They both are container based cluster manager, but their focuses are opposite. I also use "machine oriented" for Borg, and "application oriented" for k8s.
* Borg emphasize on scheduling capabilities, performance, and scalability. And various integration capabilities for wiring together the base line software foundation, and interfacing with hardwares. For example, SDN, security, name service, needs to hook up with Borg, to interact with all applications.
* K8s emphasize on modeling application semantics and how to map them into containers. K8s provides abstractions for on boarding apps and a lot of toolings.
In summary:
* K8s is considered a layer on top of Borg, if it can ever be incorporated into Google production.
* As of now, there is nothing in sight that can be considered next generation Borg. Just like there isn't any next generation Linux. Software like Borg, and Linux, can only be supplanted, not succeeded. Their vast amount capabilities defy succeeding. In other words, if one want to build a better Borg (or Linux) they are doing it wrong. K8s did not try to better mesos or open stacks, or Borg perse, it's just a different thing.
Edit: There are numerous difference between Borg and kubernetes. Send me an email listed in my profile, if you are really into the gory details. We can setup some discussion.
I think it's sad that 99% of the resources are used by hogs, though. I always thought it'd be neat to build on tiny containers; this suggests that you really can be fine with a few giant tenants and minimal colocation.
sigh some day I'll get to design my dream Borg successor, even if it's a TempleOS-like art project after I've descended into dementia.
Borg is vast. Unimaginably vast. Any engineer can start a job in the free tier for a personal web server in minutes, all the way up to saturating 100,000 cores to process some dataset. You can browse cluster job lists forever and never reach the bottom. 1% of jobs is still a huge number of different jobs.
The "hogs" are going to be jobs like web search serving, indexing and related, logs processing etc.
The "mice" are going to be the long tail of jobs. Remember that this sort of paper presents Borg as a kind of exemplar of what large cluster systems look like, but as you already observe, that's not really true. Borg is really unique to Google and Google is really unique. In particular due to the personalities of its founders and its financial position Google has a massive long tail of products and web sites that don't see much usage, or see decent usage by non-Google standards but which gets lost in the noise of search, ads, YouTube etc.
So the hogs vs mice phenomenon is telling us more about the nature of Google and how it does projects than something fundamental about job scheduler systems.
We might have something to show.
It seems that K8s takes networking under its purview, Borg does not and leaves it to the client (another comment here). K8s also implements StatefulSets for storage.