A full decade since we took the 'it works on my machine' excuse and turned it into the industry standard architecture ('then we'll just ship your machine to production').
Well, before Docker I used to work on Xen and that possible future of massive block devices assembled using Vagrant and Packer has thankfully been avoided...
One thing that's hard to capture in the article -- but that permeated the early Dockercons -- is the (positive) disruption Docker had in how IT shops were run. Before that going to production was a giant effort, and 'shipping your filesystem' quickly was such a change in how people approached their work. We had so many people come up to us grateful that they could suddenly build services more quickly and get them into the hands of users without having to seek permission slips signed in triplicate.
We're seeing the another seismic cultural shift now with coding agents, but I think Docker had a similar impact back then, and it was a really fun community spirit. Less so today with the giant hyperscalars all dominating, sadly, but I'll keep my fond memories :-)
Some of those talks strangely make more sense today (e.g. Rump Kernels or unikernels + coding agents seems like a really good combination, as the agent could search all the way through the kernel layers as well).
Funny comment considering lightweight/micro-VMs built with tools like Packer are what some in the industry are moving towards.
"Ship your machine to production" isn't so bad when you have a ten-line script to recreate the machine at the push of a button.
Wonder when some enterprising OSS dev will rebrand dynamic linking in the future...
I don't care about glibc or compatibility with /etc/nsswitch.conf.
look at the hack rust does because it uses libc:
> pub unsafe fn set_var<K: AsRef<OsStr>, V: AsRef<OsStr>>(key: K, value: V)
So what do you do when you need to resolve system users? I sure hope you don't parse /etc/passwd, since plenty of users (me included) use other user databases (e.g. sssd or systemd-userdbd).
I guess what bothers me is the software authors who don't think this through, leaving applications non-functional in these situations.
At least with Go, if you do CGO_ENABLED=0, and you use the stdlib functions to resolve user information, you end up with parsed /etc/passwd instead of shelling out to id. The Go stdlib should maybe shell out to id instead, but it doesn't. And it's understandable that software developers use the stdlib functions without thinking all too much about it. But in the end, simply advocating for CGO_ENABLED=0 results in software that is broken around the edges.
Moving from linking stuff in-process to IPC (such as systemd-userdbd is promoting) _seems_ to me like a natural thing to do, given the nastiness that can happen when you bring something complex into your own address space (via C semantics nonetheless). But I'm not very knowledgeable here and would be interested to hear your overall take.
For example, you technically can't sandbox any app with NSS/PAM modules, because a module might want to send an email (yes, I saw that in real life) or use a USB device.
NSS/PAM need to be replaced with IPC-based solutions. systemd is evolving a replacement for PAM.
And for NSS modules in particular, we even have a standard solution: NSCD. It's even supported by musl libc, but for some reason nobody even _knows_ that it exists. Porting the NSCD protocol to Go is like 20 minutes of work. I looked at doing that more than once, but got discouraged by the other 99% of complexity in getting something like this into the core Go code.
I think it’s laziness, not difficulty. That’s not meant to be snide or glib: I think gaining expertise in how to package and deploy non-containerized applications isn’t difficult or unattainable for most engineers; rather, it’s tedious and specialized work to gain that expertise, and Docker allowed much of the field to skip doing it.
That’s not good or bad per se, but I do think it’s different from “pre-container deployment was hard”. Pre-container deployment was neglected and not widely recognized as a specialty that needed to be cultivated, so most shops sucked at it. That’s not the same as “hard”.
Minus the kernel of course. What is one to do for workloads requiring special kernel features or modules?
I sort of had the problem in mind. Docker is the answer. Not clever enough to have inventer it.
If I did I would probably have invented octopus deploy as I was a Microsoft/.NET guy.
Good luck convincing people to switch!
Using it, solving problems with it, and building a real community around it tend to make a much greater impact in the long run.
That means unlike Gentoo, I've never dealt with a "slot conflict" where two packages want conflicting dependencies. And unlike Ubuntu, I have new versions of everything.
Pick 2: share dependencies, be on the bleeding edge, or waste your time resolving conflicts.
Absolutely not. Nix and Guix are package managers that (very simplified) model the build process of software as pure functions mapping dependencies and source code as inputs to a resulting build as their output. Docker is something entirely different.
> they’re both still throwing in the towel on deploying directly on the underlying OS’s userland
The existence of an underlying OS userland _is_ the disaster. You can't build a robust package management system on a shaky foundation, if nix or guix were to use anything from the host OS their packaging model would fundamentally break.
> unless you go all the way to nixOS
NixOS does not have a "traditional/standard/global" OS userland on which anything could be deployed (excluding /bin/sh for simplicity). A package installed with nix on NixOS is identical to the same package being installed on a non-NixOS system (modulo system architecture).
> shipping what amounts to a filesystem in a box
No. Docker ships a "filesystem in a box", i.e. an opaque blob, an image. Nix and Guix ship the package definitions from which they derive what they need to have populated in their respective stores, and either build those required packages or download pre-built ones from somewhere else, depending on configuration and availability.
With docker two independent images share nothing, except maybe some base layer, if they happen to use the same one. With nix or Guix, packages automatically share their dependencies iff it is the same dependency. The thing is: if one package depends on lib foo compiled with -O2 and the other one depends on lib foo compiled with -O3, then those are two different dependencies. This nuance is something that only the nix model started to capture at all.
The rest of your endorsement of NixOS is well taken, but this is a silly distinction to draw. Dockerfiles and nix package definitions are extremely similar. The fact that docker images are distributed with a heavier emphasis on opaque binary build step caching, and nix expressions have a heavier emphasis on code-level determinism/purity is accidental. The output of both is some form of a copy of a Linux user space “in a box” (via squashfs and namespaces for Docker, and via path hacks and symlinks for Nix). Zoom out even a little and they look extremely alike.
Unpopular opinion, loosely held: the whole attempt to share any dependencies at all is the source of evil.
If you imagine the absolute worst case scenario that every program shipped all of its dependencies and nothing was shared then the end result would be… a few gigabytes of duplicated data? Which could plausible be deduped at the filesystem level rather than build or deployment layer?
Feels like a big waste of time. Maybe it mattered in the 70s. But that was a long, long time ago.
Nix and guix sort of move this into the source layer. Within their respective distributions you would update the package definition of xz and all packages depending on it would be rebuild to use the new version.
Using shared dependencies is a mostly irrelevant detail that falls out of this in the end. Nix can dedupe at the filesystem layer too, e.g. to reduce duplication between different versions of the same packages.
You can of course ship all dependencies for all packages separately, but you have to have a solution for security updates.
pnpm fixed it exactly the way you describe though: content-addressable store with hardlinks. Every package version exists once on disk, projects just link to it. So the "dedup at filesystem level" approach does work, it just took the ecosystem a decade of pain to get there.
Much harder to get reproducibility with C++ than JavaScript to say the least.
Honestly, I've seen projects that do this. In fact, a lot of projects that do this, at the compilation level.
It feels like a lot of the projects that I would want to use from git pull in their own dependencies via submodules when I compile them, even when I already have the development libraries needed to compile it. It's honestly kind of frustrating.
I mean, I get it - it makes it easier to compile for people who don't actually do things like that regularly. And yeah, I can see why that's a good thing. But at the very least, please give me an option to opt out and to use my own installed libraries.
In other words, the Microsoft Windows update process as applied to software development.
If you have adopted a bad tool then people are likely to want the bad tool in more places. This is the opposite of a virtuous cycle and is a horrible form of tech debt.