Kubernetes: The Documentary | Hacker News Reader

Kubernetes: The Documentary

youtube.com

263 pointswallflower143 comments

I love Kubernetes, but this documentary makes me sad. The story of PaaS is absolutely worthy of a documentary, but to imply that Google is the protagonist is just pandering to the "rich kid" who brought little more to the table than money and influence.

Here's a better sketch for the story. It starts with Ruby, itself a game-changing language, that inspired a new culture of developer-centric best practices like Rspec's TDD and Bundler's lockfiles.

Ruby made the first PaaS possible, Heroku (a YC startup), which itself inspired a whole new way of thinking about software. I don't think enough people realise just how influential Heroku's 12 Factor App manifesto[1] was, in fact I'd argue it's the real hero in this whole story. The 12 Factors are just text, it's not even software or hard and fast requirements, they're merely best practices for the entire experience of writing, collaborating, testing, deploying, scaling and maintaining software. The 12 Factors aren't just about pushing your app on Kubernetes or whatever, they're just as much, if not more so, ideas that make _you_ a better engineer. Heroku made money for sure, but they did something beyond money, they upped the level of our craft as a whole.

The central concept in Heroku is the dyno. If you follow the 12 Factors, then you magically get almost infinitely scalable dynos. It plays right into the heart of the new gold rush world of startups, where anybody can have an app idea and leverage the global marketplace with the abstractions of datacentres and the cloud.

Docker is the open source dyno.

So many PaaS projects apart from Kubernetes have risen and fallen around Docker. Deis, Flynn, Dokku, Swarm, to name but a few. So many of the core concepts of Kubernetes were innovated and explored, and arguably mastered, before Kubernetes came on the scene. Kubernetes is a success not because of its innovation, but because it's backed by The Rich Kid. The world was already heading in this direction, Google just threw big numbers at it to make sure they took the biggest piece of the pie.

1. https://12factor.net

I've been involved in multiple migrations to public cloud infrastructure (some were unnecessary, but such are the trends...), and without failure, the 12 factors manifesto shows up in the very first days of the discussions as the first target to reach before anything else is doable.

I very much enjoy it and I'm glad whether I see someone mention it or I can spread and teach its wisdom.

This is an interesting story but it’s nothing more than fan fiction. Please provide some reason for your assertions.

While the entire comment could be contradicted, a quick example is that google’s SRE book details much more developer strategies, including from a massive scale and enterprise perspective, than 12 factor web app. This book was assembled but industry experts with experience from Borg, with borg emerging from the global work queue — decades before heroku existed.

> google’s SRE book details much more developer strategies [...] than 12 factor web app

I would argue that's precisely the problem. Google are clearly capable of doing complex things, but inspiring and nurturing an entire cultural shift is a different category of "complexity", which in the story of Paas, Google did not achieve, they merely exploited it.

You seem to be arguing against yourself here though. Kubernetes is successful, and it doesn't precisely fit the mold of the 12 factor app. It fits an overlapping but distinct, google style, development model. They successfully achieved that shift.

I think you're arguing that that's a bad thing. But it's a thing that very much did happen, because kubernetes is successful.

My thesis isn't that Kubernetes isn't good, nor that it isn't successful (I use it, recommend it and am deeply grateful for it). Rather I'm disappointed that the documentary doesn't pay respects to the milieu that primed the world for Kubernetes, which to me makes it seem like Google also inspired the mindset and demand for something like Kubernetes, which I personally believe is inaccurate and more importantly unfair on the many innovative projects upon whose shoulders Kubernetes stands.

Didn't Google's Borg precede Heroku by years? Surely its unfair to claim that they "brought little more to the table than money and influence"

I think it's clear in everyone's minds that right from the start Kubernetes was making Docker a first class citizen right? Borg did not use Docker. Borg may well be the technical ancestor of Kubernetes, but Kubernetes owes significantly more to the paradigm shift and cultural revolution that Docker brought. Therefore I believe it's more relevant to consider Docker the inspiration and ancestor of Kubernetes.

Borg didn't use docker containers, but does use containers, they're just a form of containers that are propriety and predate docker by a decade (but like they're functionally really similar to docker containers, at least from the perspective of the orchestration layer).

So docker is maybe the thing that allowed kubernetes to catch on, but that's just because the external world was catching up to what borg had been doing for more than a decade, and so kubernetes was Google sticking a decade or more of expertise in container orchestration into the OSS world.

I just don't think it's fair to define Docker by merely its ability to isolate resources (that which distinguishes containers from VMs). Docker is an entire ecosystem of best practices, standards, tooling, packaging, documentation and community support.

Google caught up to Docker not the other way round.

> Docker is an entire ecosystem of best practices, standards, tooling, packaging, documentation and community support.

When kubernetes was released, the majority of that ecosystem didn't yet exist. Kubernetes influenced many of those best practices, standards, and tooling. So in shaping the ecosystem, yes, docker was catching up to Google. Like I think this would be a reasonable argument if Kubernetes was released today, but it was released only about a year after Docker was.

(heck, iirc, even docker as a tool was originally built on LXC, which I think google functionally upstreamed as part of borg hypervisor stuff, but I could be mistaken there. I do know that LXC was based on Google work though: https://www.nextplatform.com/2016/03/22/decade-container-con...)

Edit: Ok yeah I think I figured it out, cgroups is the borg upstreamed feature (https://news.ycombinator.com/item?id=9506800), LXC and evenutally docker are built atop cgroups, etc. etc. So the entire modern containerization infra is built atop a feature borg upstreamed.

We might be going in circles a bit here. Sorry if I'm misunderstanding something. Docker was released with Dockerfiles, registry support (maybe not Docker Hub, I don't remember), a generous CLI; `start`, `build`, `ps`, etc. So I would very much say the majority of the ecosystem existed before Kubernetes.

Yes LXC and cgroups were Google contributions before the existence of Docker. But jails and containers in general were also general ideas before LXC and cgroups, I think most notably in BSD.

I feel like we must have had different experiences of that period 8 or so years ago. I distinctly remember the hype and bandwagon feeling of Docker's release. I remember various projects trying to capitalise on Docker to make PaaS-type projects; Swarm, Deis, Flynn (even my own Peas project), before Kubernetes came along. It seems so clear to me that Google were reactive rather than proactive. I really don't think Kubernetes would be a thing without Docker.

I don't really dispute much you say here (although I'd argue that calling Dockers API an "ecosystem" and "best practices" is a stretch, do you write your dockerfiles the same way you did today as in 2013?). Like yes, k8s had to come after docker. What I dispute is how any of that is the spiritual successor of heroku (or GAE or any other PaaS).

And even then I think you understimate googles contribution of "we've literally been doing container orchestration for longer than anyone else, here's tooling that is built with lessons we've learned in mind". That's far more than just "money and influence", and I don't think it's "exploitative".

That's a very fair criticism of my use of "exploited". And also I accept that "ecosystem" could be a descriptive stretch for Docker's release day.

I think in the same way that you're concerned I'm misrepresenting Google, I'm concerned that the documentary misrepresents the landscape of developers and projects that set the scene for Google's Kubernetes. Yes Google made significant contributions to the story that led up to Kubernetes, but crucially, I believe they didn't nurture the culture, they didn't create the mindset that makes developers (and managers) ready to think in a containerised way. And I think that's important because the smaller voices that did make those innovations easily get drowned out by the might of Google.

Containers as concept predate BSD stuff, HP-UX and Tru64 were two UNIXes that had them first, following on previous enterprise computing platforms.

Not really. It's not obvious from the outside, but from a Google developer POV borg (and workqueue mostly) had everything docker had. It just achieved it differently (monrepo, standard build, custom name service, etc).

Google pre-k8s achieved "container" orchestration by controlling the whole stack in detail. K8S provided a way to get most of that with mostly arbitrary code.

You're really missing my point. It's not always enough to create a technically successful project. Docker was a _cultural_ revolution, it had its finger on the pulse, and without that revolution Kubernetes would not have the traction it does.

Indeed. The striking difference is that Google's internal culture is (?used to be, at least at the time this documentary takes place) that of: build your static binaries, perhaps bundle a couple of other files in a package (basically a tarball) if you really must, and run in the code in a "container" whose system image is not controlled by the author of the code package but by the place where you deploy the code (the compute infrastructure, ime. the Borg cell). When the admins of the compute infrastructure decide to roll out a new system image they'll let you know and you'll have to coordinate in order to maintain proper hygiene.

The public version of this embraced a fundamentally different status quo: software is a mess. People pull in dependency in many different ways. You may end up using some python code that has a C extension that in order to build requires some C library whose installation instructions are a mix of run some install script after having installed some apt package.

This idea that you build a container with a container (and accepting that the resulting mess is "ok") is indeed something that has to be granted as a form of innovation to docker. It created a whole new industry of people trying to tame the dependency hell and supply chain woes that have been just swept under a carpet.

Linux containers were important, but Docker gets a lot of credit for popularizing them. Even still, Kubernetes makes containers dramatically more useful than just running a bunch of Docker containers on EC2 instances or whatever. Not only does it schedule containers, but it enables controllers and operators for enabling increasingly higher levels of abstraction. I don’t think Heroku has any analog (nor should it because Heroku is a PaaS).

But there were/are many projects that offer those higher levels of abstraction on top of Docker; Deis, Flynn, Rancher, Fleet. I just don't think Kubernetes was that innovative in that regard. It won out because I mean of course it's a fantastic piece of software, that's a given, but the differentiating factor is Google's brand, not any particular technical novelty.

I think “primed the world” is a much more defensible position than what I originally thought you were arguing (that Heroku influenced Kubernetes directly).

Sorry if I caused any confusion, that's exactly my point. I didn't mean to imply that Heroku directly influenced the design and architecture of Kubernetes.

Decades before Heroku existed? Can you back that up?

I think GAE (Google App Engine) was in production before Heroku got anything out there...

Yes they were. But like I say, I think the 12 Factors were more important. And Heroku in general just understood hacker culture better.

GAE was also a hero, but too early for non-Googler.

I think you're very deeply immersed in the Ruby/Heroku world and projecting that onto Kubernetes and everything else. Kubernetes' derives from Google's lengthy experience with Borg, which almost certainly predates Heroku. I doubt very much that Google was steeped in Heroku/Ruby culture. Yes, Heroku dynos are similar to linux containers, but containerization (the idea of binning multiple applications onto a single operating system instance) predates heroku by at least 6 years and the idea of binning multiple applications onto a single hardware instance goes back much further.

Moreover, Kubernetes and Heroku aren't even in the same class of tools. Kubernetes is an orchestrator for containers, which obviously can be used as the foundation for a PaaS, but Kubernetes isn't itself a PaaS even when you consider the various managed Kubernetes offerings which include some higher level abstractions built atop Kubernetes primitives.

Further still, it's not like Google marketed Kubernetes to any significant degree prior to its own adoption of Kubernetes as its official container orchestration offering for its cloud (which was after the writing had been on the wall for a while that Kubernetes would become the de facto container orchestrator).

TL;DR: Just because other tools and technologies predated Kubernetes didn't imply that it was significantly impacted by them.

I haven't written Ruby nor used Heroku in many years. I think it's more relevant to say I'm very deeply immersed in Open Source. I contributed to OpenRuko[1], Deis, Flynn and even wrote my own PaaS, Peas[2] that got on the front page of HN. I've seen first hand the journey that PaaS has taken. Maybe we're getting hung up on the strict definition of PaaS here. Let's just say the thing that lets us easily scale a startup idea, like Uber, Airbnb etc. Sure the dyno/docker is the core component, but we inevitably have to deal with state, Redis, Postgres etc as well. In that sense Kubernetes is very much the Open Source version of Heroku.

I think it's fairer to say that Google _repurposed_ Borg, in order not to be left out of the changing landscape. Borg may be the technical predecessor to Kubernetes, but my point is that Borg is very much not the spiritual ancestor of the new containerised paradigm.

1. https://github.com/openruko

2. https://github.com/tombh/peas

I would be very interested to hear why Heroku and not Borg is the spiritual ancestor of Kubernetes because that claim is contrary to everything I’ve read and listened to on the matter.

I'd hoped that's exactly what comes across in my top-level comment. I've been lucky enough to be at or near the coalface of PaaS for most of its life, both as a basic engineer getting my head around deploying CRUD apps on Heroku and its ilk, and as an architect contemplating the designs of various PaaSs. My little sketch of a story is how I saw things unfold. Don't get me wrong, Kubernetes is amazing, it's everything we hoped for, I use it and recommend it. But the only thing that's innovative about it is that it's backed by a big player, so we can rely on at least a few years of active development and mindshare. But from my perspective, the great innovation, the thing that really changed the game, was The 12 Factor App manifesto. That's what deserves a documentary made about it, because it's just a text document, but it inspired so much.

And arguably borg and global work queue are predated by generic batch queue of VMScluster (and possibly TOPS-20 Galaxy, but I haven't played with those) and those are predated by IBM Job Entry Subsystem 3, which even operated in similar manner to k8s with "master" mainframe, usually a lower power model, scheduling jobs on multiple "worker" mainframes.

Except k8s is really, really not a PaaS. It can be used to build a PaaS in style of Heroku, but it's not one. It's lineage spans all the way back to IBM ASP in 1967, arguably even to 7094/7044 directly-coupled system (but that didn't have multiple worker nodes), hitting various systems on the way (TOPS-20 Galaxy, VMScluster, HPC scheduling systems for MPP supercomputers and probably many more I can't recall).

It's about efficient use of multiple machines, or as Google & ex-Google employees called it in a book released at similar time as Heroku went from private beta to GA, "Datacenter as a Computer".

12 factors were important, but they just as well can apply to noncontainerised software (arguably, they are just as valid in the ~2000 world of EJB java servers, except replace "environment variables" and the like with JNDI).

That said, Rails exploding in popularity around the same time as EC2 became an option heavily helped push the generic case out of the hell of PHP with occasional CGI (ASP usually meant you had more resources than shared hosting from hell)

I think you're missing my point. We're on HN right, where we often discuss the difference between technical merit and market fit. There's a time and a place for technically correct solutions, which Google excel at, but a successful business needs more than that, it needs to capture a zeitgeist, which Google sometimes fail to do, see Google Plus for example.

Of course the technical lineage of Kubernetes is Borg and parents (which I admit I've never heard of), but I put it to HN that Kubernetes' lineage is not what created the market fit. In fact I think it's disingenuous of the documentary to imply that (in fairness it does give time to Docker), hence why I'm so keen to give my version of events.

Your point about how The 12 Factors apply just as well to non-containerised software is exactly the beauty of them! The 12 Factors don't come from management, they're not an attempt (at least directly) to capture profit from a market. They are somehow fundamental axioms about writing software in this current era, they're maybe even self-evident, universal truths. It turns out that there is a fundamental connection between the ability for a piece of software to be efficiently scaled and for that piece of software to be collaborated on. 2 or more developer machines share essential characteristics with 2 or more scaled containers, which share requirements with 2 or more CI instances ― reproducibility, behaviour guaranteed by tests, ease of downloading, security, configurability through ENV.

You see how we felt like Heroku understood us the humble developer, it made us DevOps, all whilst helping us satisfy the bottom line of managers? Kubernetes is truly an extraordinary piece of software, but it hasn't, nor do I believe it ever will, achieve anything of The 12 Factors' lasting legacy.

It is, in my opinion, at best converged evolution. Maybe because 12 factors are such self-evident truths - though if true, I'd at best credit them as formulation into a listicle of the fundamental behaviour of JCL batch programs on JES2/3 (don't have enough experience with other systems of that class to compare, but I suspect certain commonality).

What my point is that while Heroku might be a very applicable relation to some, it's very explicitly a Platform as a Service, in a way pushing a bit back towards the model that we were escaping from (by invoking Rails name) in mid-2000 to cheap full-OS machines, from the confines "PaaSsy" LAMP.

Kubernetes didn't capture a zeitgeist by introducing any of the features (though it definitely pushed containers up into enterprise world a lot). It was in no way first, or even second, despite coming from home of linux containers. It doesn't even force you to use 12 factors - I should know something about it, having done what many called stupid, as in I did "lift and shift" of over 60 "classical" applications that in no way fit 12 factors onto it.

Now, trodding a bit further away from concrete facts and into more a world of personal opinion...

Kubernetes captured the zeitgeist by working. There was immense desire across all kinds of business, developments, etc. for something that fit nebulous label of "private cloud". Often with local deployments involved. And to do it cheaper (or at least, easier to automate) than existing "heavy" virtualisation stacks. Not everyone could use at all, or just burn money on AWS, many VPS providers were more like classic VMware virtualization clusters in experience. Some had money to burn on Heroku and actually fit running on it (or were lucky to fit in the cheap band of usage).

Thus you had OpenStack and the dwarfs: CloudStack, Eucalyptus, OpenNebula, Ganeti, with more adventurous ones building on Mesos or fully custom.

All tackling the problem of, essentially, providing simpler and faster (in turnaround time) way of handling users access to resources.

The most moneyed and successful (at least from pov of Ops) was OpenStack. And it was a hot mess. Upgrading an openstack cluster was a meme, and not a nice one. Meanwhile containers gain steam, for good or bad, and people start to move from moving from the quite bad "single app VM instances", to containerised approach (12 factors already hit harder due to the single-app VM approach popularised by AWS ASGs coupled with cloud-init).

Into this enters kubernetes. Yes, it might seem hard to setup - but it's no OpenStack, it brings no licensing purchasing mess of VMware, it initially kills one of the more annoying aspects (though it will return) which is the overlay networking. It brings unified API, simple operating model (wish I kept the old 1.0-1.4 presentations on hand that went bottom up!). We can deploy stateless services in somewhat sensible way (and if you know how, they can be stateful too). The saner internal design and API commonality can be arguably compared to the effect of the original "heroku" command that allowed you to just deploy from a git repo - just from the point of Ops team.

By next year, we move from simple ReplicationController to ReplicaSets + Deployments, as well as gaining Ingress beta and first rush of "pluggable" componentry in the form of Ingress Controllers. It's arguably the time where you can start dealing with K8s as simplistic replacement for Heroku - except it's not some closed magic operated by the Wizard of Oz, but something you can deploy on bunch of Raspberry Pi in middle of presentation at a conference after suffering a long trip (we even forgot, iirc, to bring our planned kube-master machine and had to improvise). Early versions of TPRs (priors of CRDs) make me give what were probably some of the first talks about operators (before CoreOS people coined the term) - but I used the same terminology as K8s used (Controllers).

By 2017, the recommended way to deploy openstack starts to be "deploy openstack on top of k8s", and DC/OS starts to visibly lose. K8s makes certain things forced upon you, but you find that it doesn't matter and tends to make sense (Pods having one IP, etc.). Secondary/Tertiary adoption follows, bad memes start about kubernetes (like you only need it for scale, that it's cloud specific, etc. etc.) but it doesn't really matter. Soon, operators start flowing in, thanks to simplicity of the API model (this evolves later to things like Crossplane and even separating kube-apiserver into standalone program, to run without rest of k8s). K8s manifests might feel like JCL, but you don't care because of how bloody more complex the same things used to be before.

Heroku might have provided easy way to deploy constrained application for a developer, but, at least in my experience, k8s became the lever with which I could move a lot of servers with little effort, whether to run large batch jobs, complex meshes of microservices, CI build jobs, or build a heroku-style PaaS. As developers learnt to follo 12 factors, the job became easier.

P.S. Funnily enough, probably because accidents of history kept me on the Ops side, but IME kubernetes adoption always had push from the sysadmin side, not necessarily developer side.

P.P.S. As nice as Heroku is, there's a reason why it's my main example of burning money, especially contrasting how it's billed as "cheaper than paying for an Ops engineer". In the world of global pricing, what works as cheap in Silicon Valley is... not cheap in most of the world.

P.P.P.S This kinda went ranty, I guess I should maybe write a blog post on this perspective...

Very interesting about JCL being an older precedent for the 12 Factors, I've never used it, so I'll have to take a look.

I totally know what you mean about OpenStack! So much potential, but just way too complicated. But I'm not sure it's fair to say that Kubernetes was the first project to Just Work™. I never tried Mesos, but it's always seemed more accessible to me. But I did use Deis and Flynn though, and they were a delight in comparison. Deis of course being the home of Helm, Kubernetes' now de facto packager manager.

Also don't you think that Kubernetes explicitly embraced Docker right from the start as a first-class citizen? OpenStack only ever bolted it on as an after thought. Kubernetes knew what the buzz was and made sure to let everyone know that Docker was front and centre. This is why I think it's better to think of Kubernetes as a spiritual child of Docker (and therefore everything that came before Docker) rather than a technical descendant of Borg.

Although Deis and Flynn were certainly already creaking under the weight of what it truly meant to support the full gamut of cloud features, they were heading in the right direction. Google really did knock it out of the park, which I'm grateful for, but I think they did it more with the sheer weight of money and reputation than any innovative technology. Nobody really knew the devs behind Deis and Flynn, but everyone knows Google's reputation. And so we all just came to warm our hands around Kubernetes' fire, and the rest is history.

JCL and JES2/3 provided a bunch of features (of course, very primitive compared to anything newer) that helped run programs in a world before expectations of "wasting precious disk space" on configuration or state that wasn't critical, thus it tended to operate remarkably similar to running 12 factor batch jobs - for example, you would use Data Definition (origin of Unix dd command - it was a joke on JCL syntax, iirc) to specify open files (with named handles) passed to a program (consider passing that through environment variables or parameters in 12 factor app), telling it to run an otherwise stateless binary, and possibly embedding a short command program (interpreted by it) in JCL stream. A program might be preinstalled on disk, or pulled from tape, your data might be mostly on tapes, etc., and your JCL on a bunch of cards that you submit is, effectively, your declaration of environment for it and the system operators take care it's run properly - but outside of that JCL, there might be no other state involved.

Kubernetes might seem to embrace Docker, but in reality it hid the Docker. In fact, kubernetes explicitly avoided using features from docker outside of certain basics - I'd even say it's part of what I liked the most, as I found Docker to be incredibly clunky once past "local development" stage. As early as 2016 we had alternative runtimes that took in docker images (and sometimes not even them) and ran them in completely different matter. It was just a question of time, when Docker will be eliminated from k8s stack, rather than if. The only real thing that k8s took from Docker and not somewhere else was the idea of container images as packaging format (containers having been widely used at google with LMCTFY and other runtimes before, and LXC predating Docker in wider world with even primitive package system).

Mesos had the issue that, at least in my experience, every bit of the stack was separate. Running "just Mesos" wasn't really an option, and every framework seemed to be building with its own APIs, creating a disconnected mess (mind you, my first actual use of Mesos came after already using k8s for a while). What comes as common bits (without operators) in k8s 1.4, involved IIRC 2-3 different frameworks with disjoint APIs on DC/OS. Quite a different experience, even if you used DC/OS which packaged it altogether for you. Mind you, DC/OS arguably also totally killed community around Mesos in my PoV, but that's a question to someone deeper involved in that ecosystem.

As for OpenStack vs. K8s... I can't even compare the scale difference of running "reasonably operational" k8s cluster vs. "openstack that isn't just devstack on dev's laptop". And when it comes to upgrading it really becomes unfunny.

Also, at least early on, it really didn't seem that K8s would become the half ton gorilla it is now, and while people might have not known devs behind Deis and Flynn, it was in my experience heavily the evangelism of early adopters that pushed it, later bolstered by GKE becoming a thing. Despite that and google's supposed reputation (they really, really don't know how to monetize it, instead burning it - otherwise GCP would be much more of a success commercially than it is, not third in public perception of people I talk with), alternative runtimes continued on and people still believed that DC/OS might be the winner, or that a bunch of other platforms might maybe standardize some bits around docker but no clear winner might appear. Admittedly I haven't heard of Flynn, but I do recall Deis, pre-k8s Rancher, Docker Swarm, CoreOS' Fleet, the unlamented OpenStack thingy, and others being regularly discussed.

The one thing that IMO made k8s truly groundbreaking was the kube-apiserver design, from first principles. Yes, a whole cluster is complex, but it builds from simple component ideas, with the simple patterns of apiserver tying everything together (said "simplicity" is grounded thoroughly in control theory and, I wonder if anyone noticed, symbolic AI designs like Blackboard Systems). This meant that evolving that system, first all internally through combined controller binary, then through TPRs and finally CRDs, was easy. Outwardly (there's some extension if you look close) the base k8s API didn't change since version 1.0, despite huge changes in what you could do with it. Something that I don't think any other alternative took care of, with exception of maybe generalized look at AWS APIs (if you squint to blur the inconsistency and try not to notice gaps hinting at private APIs and differences between what's doable from Console and API).

A bunch of that came explicitly from having experience of much more haphazardly designed Borg APIs and their ossification, and it sometimes showed in discussions on github repo (I recall early requests for CPU pinning being refused by understandably vague but useful explanation of how it led to broken design in borg api and that they would recommend shelving till a good interface gets designed)

Tapes and cards!! OMG that puts JCL into perspective!

Sure Kubernetes' API hid Docker, but its marketing certainly didn't. And I'm not sure it's right to say Docker was eliminated from Kubernetes, it was simply made optional. I think it's hard to argue that Docker isn't still the pioneer, like I mentioned Docker is the OG market fit which all the other runtimes are trying to emulate.

So well-designed APIs, which Mesos doesn't have, I can believe that. But I think it's fair to say Flynn and Deis put a lot of effort into theirs. I forgot about Rancher and Fleet! I don't know about their APIs. For sure Flynn and Deis never got as far as CRDs, but they still achieved those basic concepts we use in Kubernetes today; deployments, storage, config, logs, routes, networks, etc.

I'm just still not convinced that Borg did anything new or innovative that wasn't already in some Open Source project before Kubernetes came along.

The EuroSys paper on Borg is a good place to get the background on Borg, how it influenced Kube, and how it relates to other systems: https://research.google/pubs/pub43438/

Well, the issue is that Borg is not public, and isn't direct ancestor of kubernetes - a bunch of design decisions were informed by the Omega project which was an attempt to build a new Borg (for example, pluggable schedulers, a rather core aspect of k8s).

Marketing hanging onto Docker was a great marketing move, even if all that docker brought was essentially images.

As for basic concepts - I believe that the core issue is that Flynn/Deis/Heroku converge from the opposite starting point (it's arguably a common case for all PaaS).

A PaaS wants to provide those high-level building blocks to the end user, but (especially the case with Heroku and other closed source examples), it hides everything else. In a way it is an offshoot of, if much better designed usually, of the classic LAMP experience, except usually with some extras provided - and these days we get APIs instead of crappy web panel (or cPanel, in fact ^_-).

The implementation details of them aren't for end user consumption, and generally the less they leak the better.

OTOH, we have infrastucture-side systems, which start with "I have X resources and I want to utilise them efficiently". Those tend to overlap with PaaS in the sense that often you need them to implement a PaaS, but how generic they are differs.

Old MPP schedulers for supercomputer clusters and things like VMScluster generic batch queue are probably some of the most obvious intermediate systems between IBM ASP and 21st century. Some proprietary, some open source just niche. They didn't necessarily handle all those higher level ideaas like deployments (closer to basic k8s ReplicationController), often disregarded storage (single system image, cluster-wide common shared mounts, cluster-wide availability of storage in VMScluster, etc.). At the same time High-Availability focused clustering was also building up, but usually with statically allocated resources.

Generally those two branches of clustering started to combine around 2000 (possibly earlier - I am limited here by my own knowledge), with "grid computing" combining ideas of supercomputing scheduling, opportunistic clustering and reliability. It's also around that time that work that ultimately results in in components making up Linux containers starts at Google, soon utilised by Babysitter (long running stable jobs) and Global Work Queue (batch jobs) Not much is published about them. Google starts working on Borg around 2003-2004, basing on the large prior art in scheduling but adapting it to new realities of large node counts in non-HPC tasks. A lot of the work filters into upstreamed kernel components over time, providing the basis for linux containers, with Google developing "LMCTFY" at some point as container runtime. However, they do not perform anything like Docker images.

Google was not the only place that noticed such issues - we had more "cluster specific" and less generic (could be considered transitional technology) cluster managers associated with Hadoop and similar, but also several (still closed, papers published) projects from Microsoft, Alibaba, and others. Some stuff leaks, some is discussed under "how to run those jobs efficiently". 2006 comes and google upstreams some of the kernel features they developed for their system as cgroups. Linux containers project starts in 2008, IIRC clearly inspired by Solaris Zones which debut in 2005. Both still replicate VM instances, if in a "thin" manner (see also OpenVZ, Linux-VServer, BSD Jails).

Amazon EC2 starts in 2006, causing huge change in how one could run their software on the internet. Curiously enough, early days EC2 is mostly stateless, this spurs "12 factor" movement further :) In few years, first custom autoscaling, then built-in AWS ASG (2009) turn up, pushing the approach of pre-baked golden VM images with your software ready to run (further pushing "12 factors", as waiting for scripts to reconfigure the service is costly when you scale up!). Sometime around that time various PaaS services, including Heroku, start building up, requiring ways to schedule customer code across machines. Google App Engine goes GA few months after start of Heroku private beta, with somewhat similar approaches (Google, being Google, is a bit more alien with the custom datastore).

Availability of huge amounts of machines that aren't tightly controlled supercomputers spurs research into clustered resource management, especially since workload-specific designs like those centered around Hadoop are too specific, papers from places running internal proprietary schedulers hint at usability of generic ones. Sometime around 2009? (correct me if I'm wrong, I can't find the start date for Mesos as research project) work starts at UCB for a system that can run diverse job workloads on shared cluster, increasing resource utilization efficiency. I can't find anything reliable on whether the team knows of parallel work at google on borg, but Mesos is probably the first generic system of that class that is open sourced. Uses kernel parts of Borg if unknowingly, doesn't play with packaging. First publications 2010~2011. Twitter becomes one of the first non-research users, running on scale of hundred workers. CloudFoundry starts Warden around that time (unfortunately, Cloud Foundry is very... galapagos-experience for me, so no further details)

2013 is a big year, sees publication of Docker, then still using LXC as backend IIRC, which brings us the image metaphor for packaging. LMCTFY supposedly also goes public around that time (but I don't recall if it was indeed opensourced then, but timeline mostly fits). Oldest presentation discussing cluster management at Google happens at LISA'13 and EuroSys 2013, discussing Omega (not sure if they mention Borg by name). I'm pretty sure the name "borg" is in limited circulation by around that time, but details are still mostly secret. Definite cross-pollination of ideas between different groups at that time, even if most stuff is under NDA ;) Google also publishes "Datacenter as a Computer".

Cambrian explosion of container-based schedulers follows, as people liberate themselves from one-app-per-instance paradigm. (personal aside: I suspect half of it is due to finding Docker horrible in actual deployment, as was my experience in 2014 :D). I'm probably missing details on some (OpenShift pre-3.0?)

Kubernetes starts based on experiences with Borg and Omega, reusing Docker in place of LMCTFY - at least part of the reasoning is obvious, as LMCTFY (and Borg and Omega) had no means to actually provide files to run (a task that was performed by unrelated software at Google). Mesos IIRC didn't originally include anything to distribute applications either. This is (IMHO) the core innovation provided by Docker, integrating packaging into our idea of container.

By 2015 containers moved seriously into public eye instead of being research project or internal implementation detail and adoption takes steam. Kubernetes 1.0 is released, and continues at insane speed (good to read the borg paper and see their pros & cons page, as it relates to this!). Initial set of features doesn't allow replicating Heroku-style PaaS, but IngressControllers debut in 1.4 (maybe 1.3?) based on OpenShift Routes, bringing HTTP routing. Enterprising admins can use it to build a PaaS, but it definitely takes assembly.

Over the next few years tooling for deployment as well as various standard components one might need become more available. It becomes easier to assemble something resembling a PaaS out of kubernetes, but that's because other people write the missing bits (Ingress Controllers, service meshes, logging handlers, etc.). Alternative runtimes build and die in the frothing frenzy of development.

And finally we have last few years, Mesos effectively dying, and most "warehouse-scale computer operating system" tasks ending up incorporating kubernetes, including various PaaS projects moving onto it as base substrate. IMHO that's due to kubernetes API design allowing it to move at speeds that weren't exactly possible for others, and also due to its more generic design (since it came at it from "generic workload scheduler" rather than PaaS angle) meaning that it was applicable in more areas than something designed originally to run a website, for example.

Great overview, quite interesting.

Just to add a bit more of history, around 2000, HP starts pushing for HP-UX Vaults as means to also package application into production and restrict what resources they can use from the OS.

Do you have some more links? I did encounter "Vault" themed name in relation to HP-UX, but it sounded more like Trusted Solaris or similar MAC/MLS enabled unix version - but then I had literally a sentence or two for description.

Unfortunately not much much, given how HPE now kind of killed all the nice documents that HP-UX used to have 20 years ago.

This is what I can still find,

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en...

Looks like something closer to Solaris Zones (or maybe AIX WPAR?), definitely interesting addition to resource management building block of the journey!

This is amazing! Thanks so much for writing it out. It's so good to read about the same story from a different perspective. I'm only going to respond to one theme in it, I hope that doesn't come across as me not appreciating or not agreeing with what you wrote.

Docker's core innovation from a _technical_ perspective may well be packaging, but I think that really misrepresents the history. Docker, like Heroku, made ordinary developers powerful. It did that through _culture_, by speaking the right language, by knowing what developers want and what they're capable of doing.

I want to talk about Jeff Lindsay[1], the co-author of Docker. He's a prolific open source contributor, a vocal critic of VC funding (and I suspect HN) and passionate advocate of the basic building blocks of software. He created Dokku, introduced Alpine Linux to Docker, founded Flynn, and Glider Labs which innovated things like service discovery and distributed logging in Docker.

Jeff lives and breathes hacker culture. His hallmark is written all over Docker: embrace open source, provide basic building blocks and encourage innovation. He's one of us, a hacker, he builds things for the sheer joy of it. It's obvious he's not in it for the big enterprise money. I mean he's not even it for the reputation, hardly anybody knows his name, everyone just thinks Solomon Hykes created Docker.

These are the people that need to be mentioned in the story, in that documentary. The people that put things into the world, because the world needed them. I don't know the name of the person that came up with The 12 Factors, so it's the same thing, someone knew that the world needed them and just gave them to us.

Of course the story of Omega and Borg is crucial, but they were always motivated by the private interests of an unimaginably large corporation, a corporation that removed its own "Do No Evil" aspiration. Jeff Lindsay has a power that Google doesn't have, the love of software for software's sake. That is a critical part of why we have Docker, and why it became so popular. And Docker's open source status and developer community is an inescapable basis for the world which Kubernetes now rules over.

1. https://dev.to/progrium/hi-im-jeff-lindsay

Now, that's a theme much more interesting from my PoV :)

I don't actually recall that, possibly because I was heavily on the Ops side (sysadmining used to be reasonably easy money, and definitely easier than finding AI job that wasn't ML). From my pov Docker caused an earthshaking change in development process, but it was heavily due to papering over all the crazy between different languages, and of course the infamous "it works on my machine" "then we're shipping your laptop" theme.

I suspect I didn't see this aspect because for me it happened years earlier, when (from my PoV) interest in Rails helped break the monolithic LAMP domination, slowly paving over the divide of those who could afford a dedicated server and those who were stuck with shared LAMP hosting.

By the time Docker landed we had quite extensive suite of tools to maintain complex setups, Puppet was a well known name, 12 factor approaches were big in cloud space, and we had 4 years of heavy evangelism to destroy the wall between Dev and Ops, stopping Dev throwing packages over it and saying "done" while ops cried, and for Ops to get dirty with devs hand to hand. In a way, Docker pushed beyond "development-time only tool" felt like a step back, especially given how early on it was considered pretty normal to run a container, exec into it, modify, then save the result as new image.

Meanwhile, devs tended to not need k8s. Usually they could provision enough locally to run what they needed (possibly in docker containers, making life much easier), but the echoes of the thinking then reverberate IMO in the regular "you don't need k8s because you're not google" message.

Meanwhile on operations side we had the unholy mess that was in no way lessened by Docker, unless you were lucky and you could push your apps into one of those early (hell, I recall a demo written in Bash, I believe) PaaSes building on the simplicity of Docker. Some developers, and less Ops, might have been mollified with that. Meanwhile we had wildly heterogenous systems at times to wrestle together, possibly heavily disparate workloads. Docker looked like a component, but there was always aspect of the amount of work needed to make it generic enough (and I have mentioned few posts ago that running docker in prod was painful, right?). So there was slow acceptance of docker images aas way to handle packaging and distribution easier, and I recall a simple PaaS being demoed by a student literally within days of docker announcement (emulating Heroku and calling it so by name, iirc?), but still, nothing much on the system side. 2013-2014 we were trying to use docker as simpler VM (especially given how hairy ephemeral LXC at the time could be) for running tests of what we would do with automation on "normal" hosts, and reuse of docker as way of dealing with packaging/runtime problem in systems like mesos (but also hadoop/yarn) was becoming more common. It's from this angle, I feel, that Kubernetes arrived on the stage, and this is why I don't necessarily see the same involvement you describe - but that's good, because I also learnt a thing :)

I just don't necessarily feel it was an aspect driving kubernetes, but it definitely feels like it was an aspect driving docker among developers - and with k8s we (more on the ops side ^_-) could deliver such environment for devs in more reliable way.

Sorry for the late reply. I really, really appreciated your in depth replies and your different perspective. Thank you so much for that, I learnt a lot things too :)

I was never really aware of heroku.

I knew app engine and other things for much longer.

I would not subscribe to your interpretation.

Scale - How Cloud Computing took over the World.

(This is describing a book that would be interesting to read).

Chapter 1 - Cobol on the Mainframe

A description of how the world's financial infrastructure ran on mainframes from the 1970s onward.

Chapter 2 - The rise of the Internet

A quick description of the take off the internet and how companies would scale on the internet. Ops teams of hundreds and Solaris machines in data centers.

Chapter 3 - Containerization, Virtualisation and Separation of process/progams

The technological routes of virtualisation and containerisation, from the Mainframe to jails to containers.

Chapter 4 - Scaling by the majors

A description of Google, Microsoft, Yahoo and others scaled their own operations in the early 2000s.

Chapter 5 - Selling Scale

Heroku and then the birth of AWS.

Chapter 6 - Everyone in the Cloud

The growth of Cloud Computing with AWS, Azure, GCP, OCP, Hetzner, Digital Ocean etc, etc.

> The 12 Factors aren't just about pushing your app on Kubernetes or whatever, they're just as much, if not more so, ideas that make _you_ a better engineer.

This is absolutely true, you can build a well factored app with intent to run it on 1 server. I would say it's very much worth it because adhering to these best practices will not only make maintenance of your app easier but it'll make things SO much easier to move from 1 server to multiple servers later on if you need to where "servers" in this case could be a Kubernetes cluster, Heroku or anything in between.

If anyone is curious I recently put out a 90 minute ad-free video going over how to create a 12 factor app with a Dockerized app at https://nickjanetakis.com/blog/creating-a-twelve-factor-app-.... It goes over the whole original 12 factor guide from start to finish and applies it to a code base.

Even now 10+ years after the 12 factor guide was originally released it still very much applies today because the guidelines transcend technology.

I remember reading this guide so many years ago and thinking "yep, this is it, they nailed it". So much goodness has come out of the Ruby community, especially around the idea of creating high quality services and tools built from really successful applications. In this case, the 12 factor guide is an extraction of best practices after running hundreds of thousands of apps on Heroku and Rails is an extraction of building Basecamp.

FWIW — we never viewed Kubernetes as a PaaS. We were informed by both Borg and GAE (and Heroku and PCF).

The pattern that you see often with PaaS is that they work great until they don’t. When you want to do something that doesn’t fit the patterns that the PaaS was built for you are out of luck and pretty much have to drop down to VMs.

We aimed to have k8s be something that was in the middle. Generic enough to work with a large number of workloads while providing significant advantages above raw VMs (and a level of consistency across infra providers).

The tradeoff is that it is more complicated than the typical PaaS. That is the tradeoff. The goal then was to build kubernetes to be built upon. We never claimed to be solving the whole problem in one system.

This is similar, in some ways, to the design of Mesos. But to do something new with Mesos requires an entirely different “framework” that exposes its own API. You see this with things like Marathon or Aurora or Chronos. While the underlying scheduler was shared across these there was no common API and so it was difficult for users to mix/match across these.

By focusing on a consistent IaaS-ish API for k8s (and eventually, with CRDs, building extension mechanisms) we aimed to create an ecosystem of things building on k8s.

Whereas PaaS are often a framework where you put your code in and a lot of decisions were made for you, I always viewed k8s as a toolkit where you get a lot of powertools that you can use in various ways. It is a very different philosophy.

12 factor and Docker obviously play a big part of this. And those built on experience and systems that came before them. We are all standing on the shoulders of those that came before us.

For me, this API design experience was informed by my time at Microsoft. Things like VB and IE/web were very much, at that time a framework (the web is turning into a toolkit over time now). I saw, multiple times, how those things were super useful but were limited. Things like the CLR and the C# ecosystem evolved the VB model to be much more of a toolkit.

We see this also in the Java world. I’m not an expert there, but the move from app servers to things like Spring Boot also show the shift from framework to toolkit.

AWS itself is a toolkit. And IMO, the difference between something like lambda/faas and a PaaS is that lambda/faas only works when it is in context of the larger toolkit that is IaaS cloud.

(I’m one of the founders of k8s and ended up with quite a bit of screen time in the documentary. My only regret is that more people that were so important in the k8s community couldn’t be featured!)

Thanks for responding to my soap boxy opinions!

I actually wrote my own complete PaaS from scratch once[1], and contributed to a number of other PaaS projects. I totally agree that PaaS projects work great until they don't. The reality is that a PaaS has to also be a Iaas, and yes Kubernetes really hits that middle ground. Thank you so much for your part in that.

1. https://github.com/tombh/peas

Thanks for your effort on k8s, as well as for taking the time to contribute to this discussion.

> My only regret is that more people that were so important in the k8s community couldn’t be featured!

Would you like to list some of those individuals here? Perhaps they have homepages/blogs we can check out?

I hesitate to start listing names as I'm sure that I'd accidentally leave someone out. Suffice to say that so many folks had an amazing impact and I'm grateful to all of them for being part of it.

Fair enough - and thank you for taking the time to reply.

> We see this also in the Java world. I’m not an expert there, but the move from app servers to things like Spring Boot also show the shift from framework to toolkit.

Having ridden the heyday of Java application servers, I see Kubernetes as LanguageAgnosticEE, just way more complex than dealing with WebSphere or WebLogic ever was.

Arguably its inherent complexity given that it's further toward bare infrastructure from them, just from supporting other languages.

> Google just threw big numbers at it to make sure they took the biggest piece of the pie.

And, ya know, invented cgroups and everything.

If they were so innovative then why didn't they take the next step and make something like Docker?

Because Docker is also innovative?

What is your argument here? If Google doesn't innovate everything, it can't be innovative?

This is nonsense.

I was put off by your sarcastic dismissive "Ya know". I may have a controversial opinion, but I don't think it warrants that kind of flippancy.

I'm sorry - I'll try to be kinder. To explain myself, I too was put off by your dismissiveness of Googles own key contributions. But that doesn't mean I can't say that kindly.

For whatever it's worth, you have a very cogent way of making your point.

Thanks. I must admit I've learnt a lot about Google's contributions from this thread, so it's been productive overall.

In some respects K8s is a bit of a step backwards from Heroku or AppEngine. It's complicated and requires you master its learning curves to deliver benefits. This looks like a lot of cloud products which are becoming more complicated to sell to bigger companies where they have the ops teams dedicated learning these products. You get an 'easy' to setup demo with a lot of copy-paste or bash|sh but you'll need a PhD for running your own apps in production. And each product is a different learning curve and they don't always play nice.

The next paradigm is composability and simplification. I should be able to define an application, its configuration, what it needs to talk to, and level of desired spend for which availability/performance SLO and it should mostly just work. Security should just work and each perimeter is automatically configured - without managing separate configuration for admin APIs, end user auth, peer auth, k8 network policies, firewall policies, etc. I shouldn't have to care about runtime - GKE vs Cloud Run vs Lambda or optimizing Node shapes or whatever new product launched yesterday. If I push a production change there should be a consistent way of simulating, canarying, and slowing rolling that change out across all products.

Ideally, it should basically be a marketplace of commodity network/compute/storage and the providers are dynamically bidding for my business. If you can run my application as fast or as reliable with less spend, you get the business - if not the app magically moves the provider that can.

> In some respects K8s is a bit of a step backwards from Heroku or AppEngine. It's complicated and requires you master its learning curves to deliver benefits.

what, exactly, do you think kubernetes is? what is the central design philosohy of it? how does it work? can you name anything about kubernetes that makes it better or different than Heroku or AppEngine? nothing you've said gives me confidenc you can identify the most basic strengths or competencies here, that you can identify how kubernetes is different from the more prosaic & narrow platforms which came before it &nehat that meams. kis is a different take than just an app delivery platform, but i dont see that recognizance here.

to me, you are propoing we should all switch from planes to bicycles because bicycles are simpler & all folks need. this reductionism is a misservice to real debate over what & where we should go.

endless consumerization of computing, treating all computing workloads as services for petty end-users, who need dumb simple integrated ways to continue shipping legacy software platforms, bundled into fancier containers, is a dead end shithole future. by themselves, simplicity & composability buy us nothing but low-ended-ness. they reduce our ability to think & integrate & fins better cloud patterns to make software with.

thankfully i see little potential threat out there from this proposal that we make ourselves more naive. tbis beleif that we can just somehow snap our fingers & simplify & the complexity goes away doesnt seem to ne enough. what i do have faith in are better management tools & practices taht emerge atop & integrating with existing full-blooded clous platforma like kunernetes. folks like ArgoCD or Flux provide a pretty reasonable seveloper experience somewhere reasonably between Herokus bramch-based deployment model & the modern open source cloud platform.

last, but definitely not least: what specifically does your comment have to do with this link that was shared? i dont see any direct connection to the link. it seems like you just wanted to dump a generic opinion on us, to neg kubernetes broadly. how if at all have you engaged the material in thia video in your negging?

edit: very quickly to -2. im not playing kind but i think silicon deserved heavily to be challenged. if you disagree, i think words are appropriate.

I'm not dang, but I would suggest you review the news guidelines, both in respect of the content of the comment and commenting on the votes on your comment.

You should take the comment at its highest and assume the person understands K8s, rather than introducing doubt about the person's competency to speak on the subject by, in effect, cross-examining them.

https://news.ycombinator.com/newsguidelines.html

Kubernetes explicitly targets the layer below a PaaS (there used to be a nice sliding demo at Google Cloud Next 2017 in SF, with a moving slider between cloud functions and I think either full VM or bare metal).

Also, it's drastically simpler than the software it heavily pushed out, and provides a lot of composability and simplification in fact compared to what was there before - but due to needs of flexibility and not being a magical genie (or Sufficiently Advanced Orchestrator) it leaves certain things on the table.

What you describe is closer to goals of TOSCA and similar, which are supposed to be parsed by service providers implemented for example on top of Kubernetes or OpenStack or <insert your imagined platform here>.

It turns out it's non-trivial, though kubernetes pushing simple standard API interaction led to interesting development in that area.

If Heroku/AppEngine is all you need then keep using them. But comparing apples and oranges and claiming that one is a step backwards is nonsensical. One is a service and one is a software that you can take with you to any cloud provider, and the problems they solve are different in nature with slight overlap.

If I were using a managed k8s offering + a gitops ci/cd tool like flux, what would I be missing?

My problem with kubernetes is for a lot of stuff you don't know what is happening really. I recently installed cert manager and the config file alone for that resource was 5k lines generated by helm. That is just crazy. I want to know generally how that part of the app is configured (it is a user facing cert) and it isn't that complicated in the real world when you set things up with nginx.

I feel like I probably should trust the people working on these products more than I trust me securing a Linux box, but... I dunno if I'm there yet.

If you want to learn the history of Kubernetes from the people that made it, I host a weekly Kubernetes Podcast. If you watched this documentary, you're the target audience. (If you're interested in the docume

Many of the people in this documentary have been on the show before; I recommend episodes with Brian Grant, Dawn Chen, Tim Hockin and Clayton Coleman.

https://kubernetespodcast.com/

+1, great podcast. I used to listen to this one regularly. But the amount of stuff they talked about was just too great to track. So now I listen to specific topics when I may need to use in order to get a informal overview.

Who is this documentary for? Anyone who's interested surely doesn't need an explanation on what a docker container, or open source software, is

I am no expert, but I know what a container is and I use it in my daily job; same for Kube, but hearing the actual people going through it feels a bit like an history lesson.

(The documentary also touches various interesting point, such as open-source in the eyes of upper management, and so on... surely there is some better source/book/article, but I found the format pretty chill and easy to absorb)

I think this is high-level enough to appeal to non-technical people. I've shared this with some friends who have money invested in cloud companies they don't really understand, and with my family for whom even my simplified job description is too abstract to really comprehend.

I think this is good, but it makes me wish for an even broader-scoped documentary on the history of cloud technologies.

> I think this is good, but it makes me wish for an even broader-scoped documentary on the history of cloud technologies.

Honestly I would love for there to be more documentaries on the more “boring” computing related stuff too, like what are the origins of VMs.

There’s a lot of pop culture material on the founding of the big tech but very little on the boring stuff without which none of this would have been possible. And my assumptions about who invented what when are always changed by talking with people who have been around for long enough.

If there were a dedicated field of Academic History dedicated to computing that would be amazing. Just having a timeline that explains the lineage of computer technologies without it being a short paragraphs in the introduction chapter in books on more specific tools, that would be great.

I am confident you will be interested in this series of articles: “Let’s not dumb down the history of Computer Science”, https://news.ycombinator.com/item?id=25907481

The top comment links to a bunch of papers.

Kubernetes: The Documentary is for history, to tell the story.

first there was docker. but we didn't know how to use it. a tiny team of google engineers spread out across various teams saw a way to mock up something more like their internal system, to allow larger scale control, but leveraging docker-ish containers. the small disassembled team fought & was semi-surprised to see it make it through the gauntlet of corporate, to get executive approval. it shares the context of why this was so important: AWS was the 900 lbs gorilla, and the world had nothing to compete with, couldn't make a dent going it alone. the tiny assemblage of folks were still basically shocked they got approval to go ahead with kubernetes, and just in time to show it off in front of DockerCon 2014, where they released what they had & started trying to build a bigger community.

from there there's a long lesson in coallition building. it becomes a classic tale in creating more value than you capture, in not trying to be winner take all. it's a lesson in creating community to fight the existential threat of winner takes all (AWS). it's a lesson in helping a team go from pioneer stage to town planner stage, in long term planning for success.

this is a wonderful historical analysis. it is packed with countless lessons & pieces of wisdom, about software, about organizations, about open source. these engineers have such great insight into where the market was, what was happening at the time, can talk eloquently about how we were in the midsts of a "rise of devops" movement but disunified & lacking in common platforms.

this isn't a practical hands on lesson. it doesn't go into what kubernetes is, it's architecture. this is a lesson to help understand the context of how Kubernetes came about. it's here to share context. it's rare insight into a rare project that happened at a just the right moment, & was released in just the right ways. we need these kind of landmark analysis to really understand software: not just what it is, but how it came about, the world in which it came to be, and to understand how & where success comes from.

Why Google despite having Borg and know-how so early wasn't 1st to the Cloud? or bigger threat to AWS/Azure?

Google's advertising business is really profitable so anything else is secondary. Amazon came at it from a business angle. They had a ton of spare resources that they only needed once in a while and figured out a way to make use of them.

>They had a ton of spare resources

Is this true? There was the story about the resources they need for Christmas shopping that went unused for the rest of the year.

But recently somebody mentioned here (I don't remember where) that AWS was a separate business and that Amazon itself initially didn't use any resources from AWS.

That’s correct, the multi year project was known internally as MAWS - Move to AWS.

building a public cloud is more or less a rebuild of any infrastructure you currently have. it would be just unwise for a company to build their private cloud with the same trade offs as public cloud.

so while there’s the ideal state of running amazon on a large multi tenant cloud, there was absolutely no safe and reasonable path to making that happen by extending the existing infra. it would’ve also hampered aws by forcing them to focus way too large way too early.

amazon.com also was really profitable when AWS was launched. IDK, at this point I think we can just agree Google is bad at diversifying. The day their ad business dies, they're done.

It was profitable, but nothing like Google’s margins. It was still just a glorified Walmart with delivery (i.e. not much of a moat).

The fear that they were plateauing was a major motivator. Google doesn’t really face that.

That could take a ton of time to die off and they could still pivot.

But they're the least diversified, after Facebook, agreed.

Amazon.com wasn’t really profitable. They ran at low margins for years. In 2006, their profit was like 500m against 10B in revenue.

Because they were using all profit to build out EC2, fulfilment centers and fullfilment tech. It wasn't that there wasn't profit because the margins were low, there wasn't profit because they were pouring it all into different verticals and building moats.

So yeah they weren't profitable but they weren't just scraping by either. I think that's one of those things that people miss when they compare a not-so-profitable startup to Amazon. The size, scale, and huge number of product markets that Amazon plays in is really pretty astounding for a company that is only 25 years old.

> They had a ton of spare resources that they only needed once in a while and figured out a way to make use of them.

Not actually the case. AWS was a completely independent stack (from hardware on up) - we couldn't use it internally for quite some time (years); and even then, only in a limited fashion (e.g. only S3 for some time)

Bingo. There are two facts you need to understand Google's relationship with GCP. Cloud margins are the worst margins of anything at Google, so if there has to be a choice on where to spend money, it can only go to Cloud begrudgingly, or as part of some long game. Second thing you need to know is Google's internal workloads are way, way bigger than you are imagining. So if you see a headline that Google invested $100 billion in datacenters, don't imagine that all, most, or even a lot of that went toward GCP capacity. They spend that money on search, advertising, and letting an AI play ten million years of StarCraft.

They didn't realise it was a high margin business until AWS numbers were reported separately.

A lot of people used to think that Amazon.com was subsidising AWS. Financials from other VPS providers suggested this to be the case.

GCP has been in the cloud business since 2008, two years after aws launched. They clearly were in a position to predict AWS’s profit margins before AWS publicized them given they were competing for same customers.

I remember when GCP launched, we had an all hands meeting for all of AWS, which all fit in a big conference room, less than 100 people. People thought now that Google got in the business that was it. The leadership, Andy Jassy, assured everyone that our jobs were safe and we could get back to work so to say. Seems funny now, but at the time AWS was S3 and EC2 and google was ahead it seemed.

When was this? 2012?

2008 when GCP launched.

Compute Engine was announced at Google I/O 2012. GA in 2013

Google app engine was definitely NOT competing with AWS. They were not selling to enterprises.

Google is poor at support, web hosting/cloud is as much about support as it is technology

In my little experience, although GCP docs leaves a lot to be desired, the support was always very thorough and fast to answer, even for not critical/emergency situations.

They have improved a lot recently

One of the frustrating thing with their docs is that they keep changing them without versioning (last time I checked). So it’s easy to feel like you’re being gaslighted because assumptions you made on the basis of the docs could have changed and then you look stupid.

AWS being the dominant player means that if they make any silent changes there will be a SO/GitHub issue/HN thread about it. But with GCP without their pricy professional tier support you can feel a bit unsure and lost all the time about your understanding of it.

To be honest, AWS cloud support isn't that fantastic either. Responses to tickets usually come in 48 hours, if not more.

I don't know about you, but in 48 hours I've usually already built a workaround and moved on to other projects.

Not saying that AWS support is worse or even equal to GCP support, but just commenting that cloud support in general feels pretty useless and an afterthought at best.

And let's not even get started on the horrible, awful UI in AWS console.

pay for enterprise support, response is within minutes

You have to pay for support. I've found it to be one of the best. I've open a ticket in the portal with a detailed issue before I go to lunch and have a level 3 tech call me before I get back.

I've had this happen multiple times for multiple services and the person's i was able to work with had enough knowledge and access to either see detailed logs and walk me through some undocumented feature or file a bug report by days end.

I'm spoiled in that regard but I've never had non-enterpise support with anything other than the 4 hour sla.

It can really depend. I know a lot of stories of AWS support being stellar while google was bad (even discounting support for non-GCP stuff from the list).

But at the same time I can refer to enterprise-tier support at AWS, which has gone past ticket stage into "regular calls with AWS support engineers to help architect our solution" and certain critical topics ended up ghosted (my first reaction was that support had no idea about the functionality we asked about, later experienced taught me that at best, the AWS documentation lied by omission. At worst it was divorced from reality.)

One theory I’ve heard (I think from Yegge?) is that Google’s internal tooling is so good at dealing with breaking changes that they’re too cavalier about introducing them.

if you have the right company setup, headcount, and culture then you can absolutely build a world where teams can be allowed to force “perfection” over time, even with amazing levels of automation.

i think the interesting thing is that while you’d think that is the ideal everyone should strive for, the reality is that’s inapplicable or detrimental to run most businesses that way.

> build a world where teams can be allowed to force “perfection” over time, even with amazing levels of automation.

I'm confused, did you mean "without" instead of "with"?

Also:

> i think the interesting thing is that while you’d think that is the ideal everyone should strive for, the reality is that’s inapplicable or detrimental to run most businesses that way.

Why would you say that is?

i meant with. google has built some really great tools for minimizing toil when pushing major changes across a codebase. even then, there’s a nontrivial overhead that is borne across the company.

> Why would you say that is?

many/most businesses don’t have the margins/scale/time to focus on it. in a lot of businesses (amazons included) the goal for the business is to try as many things as fast as possible and scale that which works and abandon what fails. the goal of infrastructure there is to get out of the way and enable the layers above to build things as naively and simply as possible with little to no churn.

I think we see a good reason here: normal people don't think "this API is so beautiful," they think "these clowns are making me do a migration yet again."

Even if that were the case, they clearly don't want to invest any extra time or money in refactor. So much of Kubernetes' design has been problematic, yet they just limp on monkey-patching and bolting on features that take years to go from idea to implementation. If you want any major changes or simplification in K8s, wait 3 years.

A traditional FOSS project is not limited in this way. The developers often just start a new major version and overhaul everything. But they don't have to answer to "the business" or huge institutional users. Commercially driven open source will always be a product beholden to stakeholders, rather than technology designed to operate as well as it can.

The last task I want to be doing is having to migrate my Kubernetes definition files to accommodate someone's bright new idea.

You already have to do that every 9 months (or is it 12 now?) as there's no "support" after that. But the point behind major versions is you don't change your definition files as long as you stay on the old major version, which is supposed to be supported for a long time.

More like 24 no? 12 until it is deprecated then 12 of support.

> A traditional FOSS project is not limited in this way. The developers often just start a new major version and overhaul everything.

And regular users hate this.

Jamie Zawinski describes this funnily, eloquently and also offensively:

https://www.jwz.org/doc/cadt.html

> This is, I think, the most common way for my bug reports to open source software projects to ever become closed. I report bugs; they go unread for a year, sometimes two; and then (surprise!) that module is rewritten from scratch -- and the new maintainer can't be bothered to check whether his new version has actually solved any of the known problems that existed in the previous version.

But he's right, and that's why:

* regular application developers who don't care about ideology

* and mass market users

hate desktop Linux.

Oh sure, that's 100% true. Patches often are applied by packagers when the upstream won't apply it themselves. But there's a big difference between patches for bugs and poor design limping on because we can't change it because people depend on it.

Users just want stable software. Developers want software that works better and solves more problems. The compromise is for users to stick with old major versions. At some point that's no longer tenable, but at least this way the software is free. Certainly no worse than when the same thing happens but you're paying for it.

(don't click links to jwz.org, he hates news.ycombinator.com, you have to copy/paste so the Referrer doesnt' reflect this site.)

Also, Google's infrastructure is so vertically integrated that it's not obvious how to offer it others. E.g. Borg assigns 1 IP to each machine, which means every pod-equivalent has to listen on a dynamically allocated port, so Google eschewed DNS in favor of a system that more easily assigned dynamic IP:port pairs without relying on TTLs (BNS/Chubby). Google also decoupled storage from compute and had a bunch of custom databases to run in that paradigm (GFS/CFS/BigTable).

The mistake also reveals the constrained imagination: Google could only conceive of offering what they had internally to others, struggled, and continues to struggle.

Amazon didn't try - they just knew that they were good at building high scale distributed systems, and thought they could build services for others. That's what S3 and SQS and EC2 were - built from scratch, not trying to figure out how to externalize the internals.

GCP was built from scratch as well.

It's more complicated, but I think everything is at least a substantial layer on top of an internal offering. E.g. GCE runs VMs in Borg using the same physical infrastructure used for other services, but GCE has its own API and virtualized networking on top that hides Borg and was not (at the time) used for any internal services.

I’ve heard Google considers (considered?) their infrastructure prowess as a secret sauce. Could it be that they didn’t want anyone else to benefit from it?

They had years of headstart on most infrastructure technology that’s ubiquitous now, but didn’t bother to offer it to others. Perhaps they thought only Google should benefit from it, while Amazon saw a business opportunity with AWS.

I'm also curious about this.

The empirical evidence seems to point that way, a lot of people were noticing that many of the Google stuff became open source or got open source equivalents from Google (Borg/Kubernetes, Blaze/Bazel, etc.) only after a critical mass of former Google employees left Google and since they liked Google tooling started replicating it as open source software outside of Google. In some cases that open source software got traction and Google seemed to be falling behind in some areas.

Which basically tells me that in the software world, you can't really have trade secrets. I could be wrong, though.

Don't forget Prometheus (clone of borgmon by ex-SRE)

There is no way to perceive Prometheus as Google falling behind, though. It's more of a lesson about the perverse willingness of the public to subject themselves to terrible technologies that Google began deprecating years ago.

Why is Prometheus terrible? It's pretty cool.

At small scale it's fine. Its architecture totally forecloses any possibility of distributed evaluation, so there is a brick wall of scalability. Its human-readable-text-over-HTTP scraping scheme is also hilariously inefficient. It is not, in short, "planet scale".

https://www.vldb.org/pvldb/vol13/p3181-adams.pdf

Well, I get the point of Google not falling behind with their tech, but even most Google competitors aren't working at Google scale.

For Prometheus they could have just chosen to aim for smaller businesses, since that's 99% of businesses out here.

So it could just be a tradeoff.

Well, to be fair, they are offering Prometheus as a service, and they built it as a façade upon Monarch, which pretty convincingly demonstrates the superiority of Monarch.

I just find a kind of perverse humor in the fact that outside google Prometheus is viewed as alien technology - it even has that condescending name - but inside Google borgmon was considered the punch line of memes and/or a hazing ritual.

IIRC, a core point in early days of Prometheus was a greatly simplified system that built on some of the same ideas, but without the parts that made it into punch line of memes or a hazing ritual for new guys.

While the text based metrics aren't best, they are a reasonable compromise, and I found that it provides a good enough common interface where if needed I can change what collects from it later on.

Are there better options? SURE! I can, however, usually get my 80% of the problem solved with Prometheus integration and worry about scaling later.

human-readable-text-over-HTTP is planet scalable

If you look at development of lots of "Big Data" stuff (Hadoop is a good example), Google shipped papers, while FB shipped code (not entirely true for Hadoop, but much more true for some of Google's later papers).

This was, literally, one of the arguments for building and releasing kubernetes. The rise of Hadoop made it much harder to justify MapReduce being different.

If we just talked about Borg, but didn't ship code, someone else might have set the agenda, rather than Kube.

One of the stories I provided for the documentary that didn’t make it (but is one of my favorites) relates to an early 2015 discussion I had with Brian Grant (googler / architect / worked on Omega / also in the documentary).

Brian writes very succinct and clear descriptions of problems encountered inside google, breaks them down to “what worked, what didn’t work”, and then would summarize a design seed in a github comment (of which several such subsystems subsequently were implemented that have been fairly successful).

I would keep mental track of some of these comments and assign a score based on how much R&D investment at google was compressed into a 200-300 word comment. I guesstimated some at being valued at upwards of $100k a word, and jokingly asked brian if he was authorized to share $10M worth of google’s secret sauce. He laughed, but reminded me of some of the same things mentioned in the documentary (which we all knew in the project at the time) - paraphrased:

“if you can seed a community project that includes some of these patterns built on that research, and that enables $10 billion of application team value, and google can capture $1B of that in google cloud, that’s a pretty good return on investment.”

And definitely among the key contributors I worked with from google was the very human and very relatable (for HN) desire to build that shows those internal ideas off and “do a successor to borg that really takes what we’ve learned and doesn’t just copy it”.

Even if it was a business strategy to help catch google cloud up… the beauty of open source is that you can benefit from it regardless, and it moves the industry forward in a meaningful way.

> Brian writes very succinct and clear descriptions of problems encountered inside google, breaks them down to “what worked, what didn’t work”, and then would summarize a design seed in a github comment (of which several such subsystems subsequently were implemented that have been fairly successful). I would keep mental track of some of these comments and assign a score based on how much R&D investment at google was compressed into a 200-300 word comment

Most places I have seen, architects seem to spend most of their time writing out overly verbose documents, detailed specifications, making presentations etc etc. There is pressure to show you are in charge of the project and accountable for it so you end up with all these ceremonies where you create heaps of content and dump it onto the teams.

Whereas the sort of situation you described is really where the role of an architect shines, in making sense of the chaos and showing light to the teams. You often don't need more than a short message or a brief meeting to convey these ideas, but such things don't lend upward visibility to what you're doing. I've always wondered if the life of an architect in Google involves these things or is quite different from most organizations?

Google and FB (and apple, but less so?) -- have been making their own servers and hardware for years...

They protect this aggressively, and the tech is impressive.

I remember the first time I accidentally saw a custom google node when I went there to pick a friend up from there, and I had gone to his desk and I saw a really nice looking PCB on his desk he forgot about, and I exclaimed "WHATs that' -- I was pretty familiar with a wide range of server boards, and I knew that was custom immediately...

He panicked covered it and we didnt really talk about it much, but it was known really early that they were building a lot of custom kit.

~2005 maybe?

Hubris.

Steve Yegge talks about it on his podcast: https://youtu.be/9v4z46Ea35Q

Amazon actually works backwards like they say - they ask customers what they want, and build it.

Google thinks they're smarter than everyone else so they can't comprehend that developers out there in the world want different things than the developers inside Google.

You can see this permeate even in this documentary, which is ostensibly about a success - when Google actually did launch something that developers really liked - K8S.

They didn't start from developer needs or wants. They started by noticing AWS was eating their lunch, and wanting to compete with them, but realizing they couldn't win by doing it directly.

To be fair, K8s has caught on even outside of GCP.

Also, while Amazon does work backwards from customers, part of them working backwards is how to translate customer requirements into a successful business. Amazon is doing this as a for profit company, not spending money on altruistic initiatives.

E.g. purely for the benefit of customers, having AWS services be multi-cloud and open source or reducing egress cost would be best.

Because K8s, while commonly deployed on the cloud, is more of a "local cloud operating system" that you use to marshall a bunch of resources into "cloud" (and finalizing TPRs into CRDs really made it shine).

To the point that k8s is base operating system for... Chick-Fil-A restaurants, deployed locally, controlling the industrial kitchen hardware.

Borg was good enough that google engineers didn’t have any desire to use GCP and thus most of the company was not dogfooding its own product.

Additionally, the “two pizza teams” style of Amazon’s development was more amenable to the AWS model of development where each piece of Amazon could run effectively in AWS [1]. Google is structured differently, and it’s only recently that both AWS and GCP could even approach running the largest systems from a company like Google.

[1] Yes I know that AWS wasn’t built specifically for Amazon engineers, but the fact that it could accommodate them over time has been a boon for their cloud, and it gives enterprises confidence that of AWS’s strategic importance and desire to make it world class. There isn’t a large hidden layer of services that Amazon has access to that the general public doesn’t.

It is because Amazon dog-fooded their own technology stack. This forced them to have a better interface to their infrastructure, which in turn made it easy to open to the public. Google on the other hand, hid behind their own complexity and let engineers feel stupid for not knowing Borg like the back of their hand.

I think Google did .. App Engine came around 2008.

Oh I was so important, me too, I am very important and we were visionaries solving very important problems that impact millions. It turned out it was very important but it didn't seem that way in retrospect. I'm also very important but also work very well with buzzwords in a way that says absolutely nothing, except that I'm important but we totally didn't know it at the time. None of us wrote a line of code but we were nonetheless very important, and, more importantly, we're even more important now.

God even their aspect ratio has to be pretentious.

Reply on news.ycombinator.com