Containers vs. Zones vs. Jails vs. VMs (2017)
blog.jessfraz.com
blog.jessfraz.com
Amen.
Somewhat tangential note: most developers I have met do not understand what a 'container' is. There's an aura of magic and mystique around them. And a heavy emphasis on Docker.
A sizable fraction will be concerned about 'container overhead' (and "scalability issues") when asked to move workloads to containers. They are usually not able to explain what the overhead would be, and what could potentially be causing it. No mention to storage, or how networking would be impacted, just CPU. That's usually said without measuring the actual performance first.
When I press further, what I most commonly get is the sense that they believe that containers are "like VMs, but lighter"(also, I've been told that, literally, a few times, specially when interviewing candidates). To this day, I've heard CGroups being mentioned only once.
I wonder if I'm stuck in the wrong bubble, or if this is widespread.
DevOps is a recent advent, too, and sounds to me like it should be populated with folks who can participate in development and operations. Most developers I’ve ever known aren’t interested in operations.
We have people who can handle the operational stuff. They’re called systems administrators, network engineers, yes, even SREs, and other folks who are really good at understanding how computers and the Internet actually work, and a webdev bootcamp gives zero context into exactly what they do. None. Then, your ten-head startup suddenly scales to needing a physical footprint because it will literally save 80% of opex, and all your DevOps Engineers say “but why? There’s AWS,” and you’re in Medium hell for weeks arguing about it.
Apropos, if I interview you and find you have written a thoughtpiece on Medium about how “sysadmin is evolving” and it’s “time for the gray beards to learn code,” you do not get a call back. That has actually happened, and no, sysadmin is not evolving. I know staff-level “full stack engineers” who can’t tell me what an ASN is. The motions in the industry have merely made those people more in demand at a few companies and served to cement their position as “where computing gets done”.
Expect serious, existential operational threats and breaches to rise dramatically as DevOps continues to “win,” and consider it a smart strategic move to avoid DevOps culture in the long term. If you write a req called “DevOps Engineer,” I don’t even know what to say to you.
I ask because I think there’s something better. I don’t know what it is yet, but I want to find out. I’m worried about wastage in DevOps methodologies, a system where nobody is incentivized to care about the right things, going on to spook the policymakers on doing software before we find out if the DevOps and Cloud worlds, both, are objectively the best way to do software for their purposes. I strongly, strongly feel like the craft is on the wrong path, and persuasive successes in industry are getting to the right ears before we know if the discipline to efficiently handle agile infrastructure with today’s tooling is even possible. I’m not convinced DevOps will organically find the right calculus to spur the kind of systems research that took us to not only where we are, but that which will take us where we need to go.
Speaking of, I’m lazily glancing at Agile here as well but I’m not prepared for a coherent argument there, beyond pointing out that we now have better tooling for managing specifications, particularly formal and mathematical ones, than the waterfall development experiences that prompted agile thinking. We need more systems research, tinkering, rethinking POSIX, all of it.
If you’re going to be as patronizing as you are, please at least read what I’ve written and respond to it.
The fact that a jenkinsfile starts with groovy and can include N number of different languages is just the nature of the beast. There is always fragmentation in software integration, and devops is integration on steroids.
Any other methods, formal or otherwise, need to provide X value at a cost of Y that makes adoption worth it. Currently if you don't use CI/CD then the value and cost propositions of adopting CI/CD actually start to make a lot of sense if you are mature enough to accurately do cost accounting on your IT management processes.
Yes, it's true, Jenkinsfiles, Cloudformation Json and Yaml all suck to work with. And configuration management is tricky. But I know that we'll all think the same thing about any other system or approach we adopt because it'll end up being work.
CI/CD may be a trade off but it allows us to focus on business problems rather than technical ones.
That is not evolution (nor an indictment of those people, importantly). The side effect, which literally nobody is paying attention to, is a future where computing as a concept is concentrating on a few major companies. Every every every every single person who says “why would I buy a rack? There’s AWS” is furthering that outcome.
My role right now is telling a major government agency how much they’re wasting on Azure. You know, because describe-instances. It’s a lot, and I think there might be a business model in “let me at your API for a week and give me 10%.” I’d be retired by Labor Day.
Reminder: They’re sending Congressionally appropriated funds. To Redmond. And they’re not entirely sure why, in $millions of cases. Line up fifty startups that have had a Demo Day and I’d bet you’d find the same thing in fifty-one of them.
That’s the DevOps legacy: don’t mind the budget, because AWS, Azure, and GCP have our financial interests in mind and APIs are cheaper to staff than fiber. Parades of like-minded individuals came to D.C. and said “DevOps! Do it!” and the agencies are now increasingly beholden to organizations incentivized by profit and those contractors took their Innovation Award check and don’t return the “um, what now?” calls. That’s the mess I’m trying to help clean up, and it’s happening across every major governmental organ in the United States.
What "cloud" really buys you is flexibility. I also don't really miss the days of buying my own servers, lugging them into a data center, waiting for drops to be provisioned, going there late at night when there's a failure, or talking remote hands through stuff.
In my current gig we've moved from the DevOps department to the Platform department as it aligns more with what we are trying to provide. A Platform for developers.
That said we essentially can speak sysadmin and can speak developers. We trust sysadmins with network, linux image and more specialist topics. We make tools for both sides and try to make them work together often sitting in the middle and negotiating.
SREs like to build platforms for exactly the same reasons you’re touching. You sound like you’re halfway there already. I strongly suggest the Google book, with “I am not Google scale” written in Sharpie on the cover for help digesting it.
E.g: In my current case the software teams own their ops, my team doesn't ops for them.
We give them a platform of centralized logging, monitoring and etc. so that they can easily ops their services but is not my phone that rings and I am not on call. I am on call if some component of the platform itself fails.
At least my perception of SRE is that they're on call for products.
That said I would frame the work we do as “a software engineer focused on operational integration“.
That does sound like a good book and I will add to my to read list.
SREs at Google own production in a very deep sense. They are decision makers on things like when teams can deploy, how frequently, what dependencies they can use, and possibly most significantly, who gets SRE support and who has to handle their own on call rotation. They also build monitoring and reliability services and tools.
Google also employs traditional Ops people, but not as many as you might suspect. When SREs look at traditional Ops work, they see a threat to reliability and a target for automation. The mantra is that the "E" isn't for show, and that SREs are software engineers who specialize on the topic of running highly reliable services. One of the things the SRE book stresses is making sure that SRE teams aren't so bogged down in oncall responsibilities that they don't have time to work on automating their oncall responsibilities.
Might be my own bias towards the SRE word.
This isn't the origin of DevOps.
I am a developer who wants to understand networks. Can you point me to some reading resources? For now, I've just been looking at the wikipedia pages for the different protocols.
But I think it would help me to work with concrete scenarios in which you use knowledge of networks to better understand things.
I would appreciate it if you pointed me to anything you think worthwhile.
What I think is as close as we get to a canonical meaning is the meaning in which it is used by The State of DevOps reports, based on very good science and research by Nicole Forsgren et al.
They characterise "DevOps" as a transition into faster deployments, shorter feedback cycles, less warehousing of unexecuted code, and having developers have generally more insight into what's going on in the production environment.
This, of course, can (and arguably should) be done in cooperation with proper system administrators, network engineers, etc.
In other words, DevOps is not in opposition of having the right people people operate the systems.
In particular, it has nothing to do with cloudifying things. You can run a product with a DevOps approach right onto bare metal servers – in fact, there are a lot of companies doing that, for simple economical and reliability reasons.
I'm all for ranting against the cloud and the little experience people have when trying to operate systems, but blaming "DevOps" for it seems like a mischaracterisation. There's a lot of value to be had by getting more feedback from production, whether production means bare metal or virtualised environments.
You’re conflating my argument with cloud ranting and assuming DevOps methodology is the only method to acquire more feedback from production by stating your last paragraph like that. I’m saying there are potentially others, but we are entrenching on this way of doing things, and people picked this particular way of doing things and started talking organizations outside “SV” into it. That conversation gets harder a second, third, and fourth time. The prevalence of COBOL reqs should warn you of this, and what DevOps will look like in about a hundred years.
This is abjectly untrue with regards to the origins of the term - though it is the current state of the world, and your assertion about job reqs for "DevOps Engineers" is spot on.
"DevOps" as a term was coined by Patrick Debois and Kris Buytaert to succinctly refer to the concept of operations teams and development teams collaborating in a more appropriate manner than the "throw stuff over a wall" which is still common in many enterprises. It was unrelated to tooling.
We must not let vendors co-opt terms in such a way as this.
As a developer I don't want to wade into the details of systems I'm using, I want to spend my time writing code that solves the business problems I'm tasked with solving.
If there is a system that allows me to do that by abstracting away the details I don't care about, why wouldn't I use that system?
This is easier said than done.
I know a good deal about how Docker works from having hammered at it. I'm not remotely an expert in it yet, but as I go I'm learning more details about cgroups and namespaces, and this is stuff I've been able to fit in while solving problems.
Abstractions help you to not think about the implementation all the time and to have your own code work in a coherent way.
Maybe in some cases, but then I could probably make a case for that being a poorly built abstraction.
All you really need to understand about an abstraction is the required inputs, and the expected output. Having knowledge about what's going on in the black box inside doesn't really serve a functional purpose IMO. This is the very purpose of abstractions. If we sat and reflect on all the abstractions we utilize every day, we would realize we can't possible have intimate knowledge of how they all work. We just don't have enough space in our brains.
I have no idea how my OS runs under the hood. I don't care as long as the inputs I provide yield the expected output. Same goes for how my phone connects me on a call, or how my car manages air/fuel ratio in order to control engine power.
For instance, you could run your package manager across all containers to see if they have packages with known CVEs. Or manage the filesystems of all containers on the system (the usefulness of this is only clear with filesystems like ZFS and btrfs). This is effectively what you can do with Solaris Zones.
These kinds of improvements to usability aren't as sexy now that everyone is really excited about Kubernetes (where you blow away containers at whim) but it is useful for the more traditional container usecases that Zones and Jails. LXC (and LXD) is probably the one Linux container runtime that is closest to that original container model.
There's also a very big security argument -- it is (speaking as a maintainer of runc) very hard to construct a secure container on Linux. There are dozens of different facilities you need to individually configure in the right order, with hundreds of individual knobs that some users might want to disable or quirks you need to work around. It's basically impossible to implement a sane and secure container runtime without having read and understood the kernel code which implements the interfaces you're using. If containers were an in-kernel primitive then all of the security design would rest in one single codebase, and all of the policies would be defined by one entity (the kernel).
You can also do this on Linux with NixOS, where you can define the system and all containers it runs declaratively. Updating the system will update everything, including the containers (of course, you can also pin the containers or packages in the containers to specific versions).
I'm not sure this is a fair comparison but that is my impression. I could be in a bubble too.
Everything looks so simple.. and it magically runs on macOS and Windows too. No wonder people think it’s some sort of a VM.
From what I read, if one packages a Linux container for Windows, Docker does then run it in HyperV?
https://stackoverflow.com/questions/52164563/what-is-the-doc...
1) A new technology or method becomes popular. 2) Developers find new advantages in using the technology. 3) Understanding of tech and original advantage is somewhat lost.
For example: containers are now widely used as part of a scriptable application build process, e.g. the Dockerfile. There are probably many developers out there who care about this and not about how containers are run and how they interact with the kernel. And for their use cases, that is probably the thing that matters most anyways.
The problem is at least in part that the term is basically meaningless. It's too broad and flexible to be descriptive. As you quoted:
> A “container” is just a term people use to describe a combination of Linux namespaces and cgroups.
If you say that a container is not a first class object but cgroups and namespaces are, I can just as easily say that cgroups and namespaces aren't first class objects, they are just terms people use to describe a combination of system calls. It's just abstractions with different amounts of leakage the whole way down.
Also the overhead intuition exists for a reason: on both macOS and Windows when Linux containers are used, there is actually a whole VM running Linux underneath. And Windows containers come in two flavors, one being Hyper-V based, so again a VM tech comes in play.
So there are technical reasons why containers are "misunderstood", it’s because most people don’t run Linux natively, and on their stack containers are more than just cgroups and namespaces.
You can take the same container image, and run using Docker, Firecracker, gVisor, or many other container runtimes. Some of them are exactly "like a VM, but lighter".
They might not know they are, but that’s besides the point.
I actually thought that was exactly the point here...
Not only do I know several devs doing cool things with namespaces and cgroup, but I myself have done cool things with namespaces and cgroups.
I played with Docker briefly, but it has no real practical application in my line of work.
See https://www.kernel.org/doc/Documentation/cgroup-v2.txt
> "cgroup" stands for "control group" and is never capitalized. The singular form is used to designate the whole feature and also as a qualifier as in "cgroup controllers". When explicitly referring to multiple individual control groups, the plural form "cgroups" is used.
To this day, I've heard cgroup mentioned only once...
To put forth a more substantive argument, everybody has a layer of abstraction they do not peek under. You interviewed people that didn't peek under container. You went a layer deeper, but never peeked at the source tree to learn what cgroup really is. Does it really feel that much better to be one level above others?
Reminds of this xkcd: https://xkcd.com/435/
The beauty of programming is abstraction - you write a function to do one thing, do it well, and then you can abstract that concept away and reuse that abstraction. Although it's only an abstraction - a container is "like" a lightweight VM, and you can use it like that until it doesn't act like a lightweight VM. In which case, you have to dabble one layer deeper just to understand enough to know why the abstraction of your layer isn't a perfect analogy.
In which case, if you're looking for an expert in layer n, then basic knowledge of layer n - 1 might be a decent proxy for expertise
Sure. Though it's reasonable to want your level N developers to have some idea of what goes on at levels N-1 and perhaps N-2, cf. Law of Leaky Abstractions etc. It's similar to wanting your developers to be aware of their users' needs, which are level N+1 concerns.
It seems like cases where that deep knowledge is an advantage are rare but also very high value. I wonder how the EV pans out, both for individuals and orgs.
They're free to say this, but since it violates the rules of the language they're never going to get any significant level of compliance.
Yes? Because levels are finite and quantifiable.
So yes, maybe you can enumerate the levels, but at some point they become not useful to discuss. The same way that discussing NPN gates is not really useful to discussing containers and VMs.
I hope you meant have not peeked under, so far. Any obscure problem will you down the rabbit hole. If you just stop looking then how can you solve problems?
For what it's worth, one of the biggest "containerization" recommendations is to not run your database (example: Postgres) in a container, correct? Due to I/O performance decrease?
docker run --rm \
--name postgresql \
-e POSTGRES_PASSWORD=postgres \
-d \
-p 5432:5432 \
--cpuset-cpus="0-1" \
--cpus 2.0 \
-m=1024m \
--mount type=bind,source=$HOME/docker/volumes/postgres,target=/var/lib/postgresql/data \
postgres:12.2
$ pgbench -c 100 -j 2 -T 60 postgres
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 100
number of threads: 2
duration: 60 s
number of transactions actually processed: 17660
latency average = 342.761 ms
tps = 291.748482 (including connections establishing)
tps = 291.791293 (excluding connections establishing)
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 100
number of threads: 2
duration: 60 s
number of transactions actually processed: 13014
latency average = 466.928 ms
tps = 214.165822 (including connections establishing)
tps = 214.199201 (excluding connections establishing)
26% decrease, with a bind mount on ubuntu 18.04
We do run databases that do 100k`s ops/s in containers but we don't run them in kubernetes. We just mount the VM hard drive in it.
With a healthy understanding of how the individual storage engines commit to disk, upgrading, backing up, etc. can be done in parallel and without impact to a running production system thanks to the power of overlayfs.
I ran some casual tests using and found out there is a performance hit in using db binaries inside a Docker container due to Docker networking (different for different types of networking).
You generally are trusting a database to keep your data safe, so those things will contribute to data loss.
Remember the freakout about PostgreSQL not handling sync() correctly on Linux due to ambiguity in the man page? Having a networked filesystem + additional abstractions (like layers) etc only reduces data durability.
looks like docker with a bind mount has a 26% decrease in performance versus native
I can't blame them, however.
If you're a dev at a company where such misunderstandings are pervasive, and you're interested in trying to get your app running in this "Docker thing that you've heard of", you will, probably:
- need to seek out a Linux dev server because you probably don't have admin rights on your machine, and getting Docker installed onto your machine in a way that doesn't suck is more trouble than its worth,
- have engineering management that are being told that containers are like VMs, but smaller, likely from magazines or sales/pre-sales consultants,
- Have to wait days/weeks to get SSH credentials to a dev server that has RHEL 7 on it, hoping that it has Docker (a thing you've heard of at this point, but don't really know much about it otherwise),
- Have to wait even more time to get Docker installed for you on the dev server by a sysadmin that dislikes Docker because "it's insecure" or something, and
- be constantly reminded that your job is shipping features before anything else, usually at the cost of learning new things that make those features more stable
The point here is that environments like this are barely conducive for getting engineering done, let alone learning about new things. Moreover, the people that will do anything to scratch that itch usually find themselves out of companies like that eventually. It's a circle of suck for everyone involved.
So when I introduce containers (which Docker is, by far, the most common runtime, so I introduce it as "Docker" to avoid confusion) to someone who doesn't know what cgroups or namespaces are, or someone who responds with something about containers being VMs or whatever, I happily meet them where they are at and do my best to show them otherwise.
It makes it very clear what docker is, and isn't.
https://github.com/p8952/bocker
Specifically, the bocker_run function: https://github.com/p8952/bocker/blob/master/bocker#L61
E.g. see this line of code: https://github.com/p8952/bocker/blob/master/bocker#L25
"But I thought bocker implements Docker?" It doesn't, it only attempts to implement the "Docker daemon" part, and piggy-backs on Docker registries for image distribution. This is a huge part of the power of Docker, and why Docker isn't "just cgroups and namespaces". Cgroups and namespaces are a critical element, but Docker is much more than that too--it's also a set of standards for distributing and administering configuration+data for cgroups and namespaces.
That sounded wrong intuitively, so I decided to look it up: Wikipedia claims that mp3 was initially released in 1993, and the iPod was initially released in 2001, so not decades and not even a full decade.
The point stands though, that Docker didn't invent containers by a long shot; but they did make them massively more useful.
It helps focus on what Docker does provide, as you mention.
"Docker implemented in around 100 lines of bash."
This is simply not true.
Can you shed light on some of these, maybe I haven't encountered these in my day to day ? (Please note I am not talking about containers running in VMs, which apparently Docker does now).
This is both very right and very wrong.
LXC is one way to do so, runc is another way to do so, docker is a third way to do so, all for Linux. Now if you took some other OS, there'd be different solutions, each with slightly different details and thus properties, but same idea.
I mean, do you ask people what SQL is? And get frustrated if they don't start talking about MySQL specific details like InnoDB and what not?
Don't know, I feel I can't agree with you, do devs feel there's less magic involved in VMs? Honestly, I have less idea what VMs are built on top of than I do for containers.
In this context, it's not, it's specifically referring to a process running in a cgroup.
> LXC is one way to do so, runc is another way to do so, docker is a third way to do so
Docker is a suite of tools for managing running LXC or runc, both of which set up processes running under cgroups.
> Now if you took some other OS, there'd be different solutions, each with slightly different details and thus properties, but same idea
You should read the article.
Wow.
I have always wondered in which cases one would use "containers". I have asked so many Docker enthusiasts, "Why would I use containers instead of the strong process separation functionality that's built into the operating system, like users, cgroups, SELinux, etc?"
The answer has freakishly invariably been "Containers are lighter than virtual machines. Virtual machines use up all your resources. Containers can run just a single application, where virtual machines must emulate a whole operating system."
Now you're probably going, "How is that an answer to the question?"
And I know! It's not. These Docker enthusiasts have been impossible to get an answer out of. They have appeared like they're just parroting whatever they've heard from someone else as long as it seemed vaguely related to the question.
Now that I finally have an answer to my original question, it all makes sense again. And I'm inclined to agree that if you're stuck in the wrong bubble, we're both stuck in the same bubble.
Before LXC was introduced it was somewhat painful to manage multiple environments using chroot, managing network, etc.
But running the same software in a single environment wasn't always easy. You had to take care of different software versions. And it wasn't uncommon for things to break frequently because of that.
While things like python venv, ruby rvm, etc. helped dealing with it, there was no universal tool for whole filesystem environments besides virtualization.
When LXC came out, I started using it for everything. Nowadays I use LXD and sometimes Docker and it is so nice and requires minimal effort. I know that without those tools it would be very inconvenient to manage my own servers. I have separate auto-updating containers for everything and if one thing breaks, it doesn't take everything down with it. And when everything is contained and each system has minimal set of packages set up, over the years rarely anything ever breaks.
And let's not forget that these Linux features also enabled the universal packages (flatpaks, snaps, etc.) which make it easier for Desktop users to get up to date software easily.
Of course I know that it is not virtualization. But why do people say "containers are not really containers"? It still contains different environments. No one said it is about containing memory or something else.
Hell if you try to explain containers to 99% of the world programmers by saying "namespace + cgroup", I'd bet you that 0% of them will understand what you mean.
Instead, if you tell them that's "like a VM, but faster, because it "reuses" the host's kernel", you might be able to reach a substantial amount of them, and that level of abstraction would be enough for most of them to do something useful with containers.
Maybe the question you should be asking your candidates is: "How would you implement a Docker-like container app on Linux?". That's a question that specifically constraints the level of abstraction of the answer to at least one layer below Docker, and also specifies that you are interested on hearing the details for Linux.
The result is not people getting experts at all those things, but getting capable of producing something with all those things.
Obviously, to do so, people must take shortcut, and container ~= docker ~= "like VMs, but lighter" is good enough as a simplification for producing something.
Now there is something to be said about the value, quality, durability and ethics of what is produced.
But that's a choice companies make.
We have servers, they handle a lot of traffic. It's the only thing running on the machines and takes over all the resources of the machine. It will need all the RAM, and all 16 vCPUs are at ~90%.
It's running on GCP. To rollout we have a jenkins job that builds a tag, creates a package (dpkg) and builds an image. There's another jenkins job that deploys the new image to all regions and starts the update process, autoscaling and all that.
Can containers help me here?
1. Packaging and distribution- it's very easy to set up a known good filesystem using docker images and reuse that. There are other solutions- dpkg plus ansible would be an example.
2. Standardized control- all apps using 'docker run' vs a mix of systemd and shell scripts can simplify things.
3. Let's you tie into higher level orchestration layers like k8s where you can view your app instances as a single thing. There are other solutions here as well.
4. Can use the same image on dev machines as prod instead of needing two parallel setup schemes.
If you already are happy with your infra, certainly don't change it. I think once you know containers they are a convenient solution to those problems, but if stuff is setup they already missed their shot.
Like, let's say you found some binary and you don't know what it does, and don't want it to mess anything up. Is there an easy way to run it securely? Why not? And how about giving it specific, opt-in permissions, like limited network or filesystem access.
If I understand correctly, when I run a docker image on Linux then the dockerized processes's syscalls are all executed by the host kernel (since - again if I understand correctly - the dockerized process executes more or less like a normal process, just in isolated process and filesystem namespace).
Is this correct?
But how does docker on windows work?