A Linux sysadmin's introduction to cgroups

The reason we need an orchestration system for our orchestration system, is the orchestration system is a snowflake. We need an orchestration system because the container system is a snowflake. We need a container system because containers are a snowflake. None of it is really standard or easy to implement, because none of it was designed in the spirit of Unix. It was all just some random companies who threw some crap together so they could start making money selling ads or hosting services. (I don't mean cgroups, I mean the tools that use them, although cgroups are kind of warty in themselves)

All most people need to do is run a couple processes on a couple hosts, send packets between services (and networks), and have something start/stop/restart those processes. You can have all of that with, say, systemd (ugh), or three different standard interfaces that any program can use with standard libraries/ABIs. Notice I didn't say "with 5 different daemons running on 3 hosts that need complex configuration, constant maintenance, and a migration effort every 3 months".

The Open Container Initiative seems close to getting the first thing done. If we get the other two standardized, we can lose a whole bunch of the onion layers. Consequently a lot of people will need to find a new way to make money, because there'll be not nearly as much need to pay people to deal with the onion.

>"Nov 1999: Alexander Tormasov visited Singapore and proposed a new direction to Sergey Beloussov: container virtualization. He formulated three main components: containers as a set of processes with namespace isolation, file system to share code/ram and isolation in resources. Indeed it was 1999 when our engineers started adding bits and pieces of containers technology to Linux kernel 2.2. Well, not exactly "containers", but rather "virtual environments" at that time -- as it often happens with new technologies, the terminology was different (the term "container" was coined by Sun only five years later, in 2004)."[1]

[1] https://wiki.openvz.org/History

There were many efforts to get some kind of container technology into Linux, from the early 2000s (and probably earlier). VServer, OpenVZ, etc, were all trying to support full server virtualization.

Cgroups was different from most of them in that it wasn't trying to present a virtual server environment, just facilitate resource control. At Google, all the jobs we ran on Borg (the internal system that later inspired Kubernetes) knew that they were in a shared environment, so there was much less need to fake a private virtual environment. The low-level libraries linked into all Google binaries did a lot of coordination with Borg to make the sharing fairly painless. But we were trying to run tens of jobs on each of our increasingly-larger machines, and some of the batch jobs would inevitably end up hogging resources, which would hurt the important latency-sensitive jobs. CPU and memory isolation were the main things that we cared about, with network/disk as secondary elements.

For a while (in 2005-2007) in Borg we approximated this by using the kernel's "fake NUMA" support (originally intended for testing the NUMA code) to break up the system into a bunch of fake NUMA nodes, and using cpusets to reserve CPUs and memory chunks to important jobs. It was pretty ugly and rather coarse-grained, but it worked rather well and was the first widespread (running on millions of Google Linux servers) userspace system for controlling a cgroups-like system.

Essentially, cgroups piggy-backed on to the existing cpusets mechanism/API which had already been accepted into the kernel, and expanded it into a more generic way of creating a hierarchy of groups and mapping a process to a group. This was a lot simpler to get accepted than an entire virtualization system. Given that mapping, making other resource scheduling be group-aware rather than just process-aware was much more straightforward. The same userspace support in Borg that had been used for controlling cpusets worked pretty much as-is with cgroups, since the basic API was the same (just more resources were supported).

And as noted, OpenVZ and Linux Vserver were earlier container subsystems for Linux.

It basically says:

    Based originally on the cpuset system, extracted by Paul Menage
    *  Copyright (C) 2006 Google, Inc

    *  Copyright notices from the original cpuset code:
    *  --------------------------------------------------
    *  Copyright (C) 2003 BULL SA.
    *  Copyright (C) 2004-2006 Silicon Graphics, Inc.
    *
    *  Portions derived from Patrick Mochel's sysfs code.
    *  sysfs is Copyright (c) 2001-3 Patrick Mochel
    *
    *  2003-10-10 Written by Simon Derr.
    *  2003-10-22 Updates by Stephen Hemminger.
    *  2004 May-July Rework by Paul Jackson.