“It works on my machine” turns to “it works in my container” (2019)
dwdraju.medium.com
dwdraju.medium.com
I'll go as far as to say that if you want reproducible images that don't unexpectedly break without some kind of ability to trace it back to a change, always use sha256 digests in your `FROM` clauses, never fetch files directly from URLs (and/or check sha256 digests of things that you do fetch,) be thoughtful about the way your container is designed, and favor multi-stage builds and other OCI image builders to writing silly shell scripts that wrangle the build command in unusual ways.
But generally? It's still a massive improvement just having the steps written out somewhere. When I first started, it seemed the best you could get is a Vagrantfile that basically worked after you stepped on a bunch of rakes to figure out exactly what quirks you had to work around. With containers, things break a lot more predictably in my experience.
- it's usually headless
- it comes with a layer that mounts the host file system, instead of installing extensions
- better testing of that layer on all platforms, especially the ones that need to add a Linux kernel? (Windows and Mac)
- it's harder to ssh into a container, manually fix things and persist the changes without updating the Dockerfile. We can do that with a Vagrant machine.
Anything else?
Perhaps the difference is that storing your changes is an explicit operation that needs to happen consciously, and that data is usually persisted outside of the container.
That's more difficult to happen with a container and its image.
On the other hand, to have actual reproducibility you need to self build and self host everything down from base userland. However, once you achieve that, reproducible machines are one `apt install devenv` away.
What docker/containers do and excel at, compared to traditional workstation provisioning, is reduction of dependency trees via isolation. With one single user process running dependency trees are shaken and chances of accidental dependency collision drop. Does this count as solving dependency problem? Personally, I say no, but understand the other side of the argument.
And being able to make hermetic environments for integration tests used to be almost impossible, and today, depending on your stack, it is trivial with libraries like testcontainers.
It really does come down to the concepts that underlie OCI images and Dockerfiles. Why?
- Immutable root design promotes hygienic containers that keep state only in mounts. This makes upgrading, backup, etc. of containers very easy. Containerized software can be written around the assumption that this is how things work, by having a way to e.g. run database migrations at startup.
- Composable images: Rather than having every environment ever made try to perfect making a reproducible Go development setup, you can just use FROM statements to use other images. With multi-stage builds you can even compose multiple images together to an extent.
- Single-process design: Prior to containers, most VMs operated like full machines that could standalone, with just emulated or paravirtual hardware. Containers are designed around running a single process. In practice, this dramatically lowers the complexity of containers and their lifecycle. Of course, you Can and sometimes even will need to have multiple processes in containers sometimes, but the general mantra holds well.
- Composable containers: Because of the previous point, it's natural to compose containers too, using features like Podman or Kubernetes pods, or Docker Compose. Containers in the same pod can share resources that would otherwise be isolated across typical containers, such as a network namespace. This makes for concepts like sidecars that allow composable functionality, and the ability to e.g. re-use PostgreSQL and Redis containers.
- As a bonus, these approaches avoid plenty of typical pitfalls of configuration management systems often used with e.g. Vagrant such as Ansible. Because the environment is built up from scratch to your specifications, rather than working backwards from a mutable Linux system and trying to bring it to match a certain specification. Some of the problems that can lead to configuration drift still exist, but they are somewhat contained, especially if you use tags and/or image digests to pin your images.
Virtual machines are technically fine, but containers elegantly offer solutions for both development and production deployment issues that VM setups never really did prior.
Indeed, you could make something like testcontainers but for VMs... but then what do you use in place of OCI images and Dockerfiles for the base images? And you need to handle networking between VMs and the host, probably across multiple different VM runtimes. Eventually you realize you're just making worse containers!
Much of what you talk about is tooling and patterns around containers, not containers themselves, such as networking and volume mounts. I would also point out that there are better options to Dockerfiles for containers these days as well, that unlock the real potential of buildkit.
We moved from "here is an (incomplete) description of what I did", which rounds to "please guess all the previous stuff that I did to my environment, before even starting to work on getting my instructions to work" to "download this binary and execute these 5 lines". The incompleteness problem of the description of _the actual thing you want to do_ hasn't gone away, but the largely not even mentioned description of where you have to do the thing... that has been replaced by Dockerfiles
If you think Dockerfiles are great, then you haven't hit the day 2 problems yet or explored the better alternatives for defining containers.
Also, the whole point of the original post is to point out that what you are saying is not magically true for containers
The reason why containers are good is entirely about the concepts, NOT the actual engine or technologies. Again, you can literally run OCI images in a virtual machine; Ignite will run them in a KVM. That's as literal as you can get.
Docker existed and did a lot of the same things it does today in 2013.
The ONLY thing docker did was make a small amount of questionable UX around the low-level container primitives. They did not invent anything, definitely not containers. I could share VM images before Docker, nothing new there... the concepts all existed before containers.
You can literally run... anything... on a VM, why make a point about being able to run OCI images? This is like a no-op.
I feel like we took Ansible roles (for example, it could be Puppet, CFEngine, whatever) and spread them in uncountable and not reusable Dockefiles and bash scripts (oh the irony). But we still have the config managers too, because who could have imagined, you still have to have bare metal underneath.
Docker (like every other tool before it) started nice, clean and simple, because it didn't cover all the real needs. As those where added on we ended up with tens of tools on top of k8s and now here we are in yaml hell with version compatibility still not solved. And a new generation will come along and repeat it with another set of tools becase "this time it will be different". You can not get rid of complexity, you can only move it from here to there. And if there == YAML then may God have mercy on you.
They both achieve the same aims - reproducible setup - but (on Linux) a container is uses less resources than a VM, so it's lighter to run them both side by side. This matters for developer machine setup.
When it comes to production deployments, I haven't even started looking at k8s because I don't need that level of complexity.
No, you cannot remove it, but you can definitely add more. Every time I have to work with the modern Javascript tooling upon tooling upon tooling, I want to punch a baby rabbit.
If it worked, it worked, though. I'm not arguing with results.
What we actually did, back in the day, was host our own yum repo and build RPM packages. For the most part, the only differences between the nodes were the installed packages. But having to define 100s of node types in puppet and force every developer to know at least some puppet, and troubleshoot failed runs, and understand RPM packaging foot-guns. Like, if you screw up your package uninstall scripts or file lists you can make it impossible to remove a package. Or you can leave unwanted files behind. Or two different developers can try to "own" the same file, so one of those RPM installs is gonna fail.
Fundamentally, this also leans towards long-lived servers that you tend to and care for. We had teams with long-lived servers that deployed incremental app updates every few days. You know how many times we tried to build a node from scratch, only to find out that it failed to build? Or worse, it completed the build but the app is broken in some way? Eg: some historical package gets removed but leaves files behind, and this files are actually useful, so everything still works. But on a clean install it doesn't, so you end up on a sev1 incident using strace, ldd, hunting in logs etc etc.
You also wouldn't believe the number of incredibly stupid screwups that broke dozens of servers at a time when developers were building their own RPM packages and using Puppet to deploy them. This was the cause of most of my work as an SRE.
What made docker so good was that developers weren't given the ability to package up something dangerous into an RPM that gets installed by root, potentially fucking up the host.
Also, building a container either works or doesn't. This forced our developers, for the first time ever, to know 100% of the steps required to get the software to work in isolation.
And my Alpine container can peacefully coexist with your Ubuntu container on a CentOS system without filesystem clashes.
And instant rollback meant running the previous container tag.
Honestly, the puppet > docker change for applications was the biggest leap in reliability and simplification that I ever experienced.
Building an image from the ground up is the other direction. The difference is subtle, but if you have an exact known starting point, then declarative configuration management is not only unnecessary, it's downright worse than just running a few commands to get to the desired state.
The proof for me is in the pudding. Container building is a superset of configuration management. You can definitely use Ansible or what have you to build a container root image, but it would be long, overcomplicated, and more fragile than just starting from a known state and mutating from there.
I recently contributed to the Linux kernel and they often get irate over the contributor not following manual steps. They could automate a lot of it with basic features like CI, and your suggestion that it is easy to make things reproducible if you just follow a list of instructions is part of this problem. "Just follow a list of instructions" will never be a solution to a bad workflow, and it is no replacement for a good methodology.
If you do not force best practices in the tool, it permits mistakes. Something you probably don't want to allow in a tool that builds software and manages the software supply chain. Docker doesn't provide a language for describing things correctly, you can make all the mistakes you want to make. Nix, for example, is a domain specific language which won't permit you to commit crimes like not pinning your dependencies, at least in 2023 with pure evaluation mode (on by default in flakes).
> They list the steps that you need to build a working image.
No they don't. They typically list the steps that you need to build a working image yesterday, not today, or in 5 years, which is a very important thing to be aware of, otherwise you might assume the instructions in the Dockerfile were crafted with any intention of working tomorrow. There's no reason to believe this is true, unless you know the author really did follow your list of suggestions.
Nix crashes when you make mistakes in your .nix expression. `docker build` won't crash when you make mistakes in your build, it is unaware and doesn't enforce a reproducibility methodology like Nix outlines in the thesis, an obvious example being the unconditional internet access given by the Docker "sandbox".
Docker does not make distinctions between fetching source code and operating/building that source code. In Nix these happen in two separate steps, and you can't accidentally implement your build instructions in the same step as fetching your program, which would otherwise lead you to execute the build on untrusted/unexpected input. This is just one part of the methodology outlined in the Nix thesis.
TL;DR Nix doesn't suggest you follow a list of instructions to make things reproducible, it just doesn't permit a lot of common mistakes that lead to unreproducibility such as not pinning or hashing inputs.
Manager: "Fine, then we will ship your machine"
And thus docker was born.
I'm guessing it may be one of these things where it only starts to make sense after something has matured enough to warrant being replicated en-masse in a data-center environment.
Then again, I tend to live in a world where I'm testing the ever-loving crap out of everything; and all that instrumentation has to go somewhere!
Simple answer: with configuration. In Docker Compose or Kubernetes. Less often in Mesos. Maybe I want to run it on a fleet of VMs, maybe on bare metal.
> and you're doing what you'd be doing on VM anyway
Right. But with containers I can have different apps using different dependency versions. Some things use nginx, some use some other web server. Some things run with node 12, some with node 16, some things use MySQL, some other things need Postgres. This is so easy with containers.
And like, that's just minimum: I generally care deeply about the filesystem that PostgreSQL is running on and I will want to ensure the transaction log is on a different disk than the data files... and I now have to configure all of this stuff multiple times.
At some point I am going to have to edit the configuration file for PostgreSQL... is it inside of the container? Did I have to manually map it from the outside of the container?
The way you access PostgreSQL locally--maybe if you find yourself ready to add a copy of pgbouncer, but also just to run the admin tools--is via unix domain sockets. Are those correctly mapped outside of the container, and where did I put them?
I honestly don't get it for something like PostgreSQL. I even use containers, but I can only see downsides for this particular use case. You know how easy it is to run PostgreSQL in some reasonable default confirmation? It is effectively 0 commands as you just install it and your distribution generally already has it running.
And how is that different from running directly on the OS?
this makes sense when you're trying to be deployable universally, but it increases the amount of onboarding that someone needs to receive before they're proficient with the container system; onboarding they may have not actually needed to get the software working and well understood, simply 'docker overhead'.
from personal experience : i'm a long time old linux person, the insistence on going 'all in' on Docker (or whatever) just to run a python script that has two or three common shared dependencies gains me nothing but the hassle of now having to maintain and understand a container system.
if you're shipping truly fragile software that is dependent on version 1.29382828 rather than version 1.29382827 then I understand the benefits gained, but just to containerize something very simple in order to follow industry trends is obnoxious, increasingly common, and seemingly has soured a lot of people on a good idea.
p.s. : I can also understand the idea of containerizing very simple things as parts of a larger mechanism; I just don't get it with the promise that it'll reduce end-user complexity, it isn't that simple.
1. I can start it and stop it automatically with Compose. That's a big increase in ergonomics.
2. I don't have to write my own scripts to set up and tear down the database in the dev environment. The Postgres image + Compose does all that for me.
3. Contributors who don't know much about Unix or database admin stuff (data analysts learning Python & data scientists contributing to the code) don't have to install or mess with anything. It works on everyone's machine the same way.
Volume and port mapping are basically trivial concepts anyway. There's been zero downside for me in using it, even though I personally have the skills and knowledge to not "need" it. Why would I go without it? It saves me time and effort that could be significantly better used elsewhere.
However, we live in the world where the choice we have for new hires is: a) teach them all of those OS fundamentals, b) give them Docker.
This doesn’t mean we shouldn’t strive to teach said new developers all those underlying concepts, but when we talk about training juniors and have them contribute relatively quickly, it’s a much smaller surface area to bite through.
That's what I said.
Furthermore, let's be pragmatic. The operations team needs to know, yes. We want new hires to contribute and feel productive. They'll naturally learn while working on software. A junior person can contribute almost immediately with a limited surface knowledge.
Less friction: no need to understand systemd/upstart/rc what have you, /etc, /opt, /usr, mount, umount, differences between various distros, build-essentials, Development Tools, apt, dpkg, rpm, dnf, ssh keys, ...
The right time will come but give them an easy way in. Containers provide exactly that.
All they need to know: it's somewhat isolated so under normal circumstances whatever you do in the container doesn't affect the host, how to expose ports, basics of getting your dependencies in, volumes, basic understanding of container networks - for things to talk to each other they need to be in the same network. Enough to start.
- Where is the data stored? Your compose file will tell you.
- Is the configuration file in that container or mapped? File tells you. (If you didn't map it, it's container-local)
- How do you get to it with admin tools? If you mapped those sockets outside, file tells you.
Solution: config file for the config files.
The amount of work you are complaining about is, objectively, trivial. It's no harder than learning how to deploy on another distro or operating system, except in this case, your new knowledge is OS-agnostic.
As a user, this is why I love Docker. The configuration is explicit and contained, and it's well documented which directories and ports are in play.
I don't need to remember to tweak that one config file in /etc/ which I can never remember where is. Either it's a flag or it's a file in a directory I map explicitly. And where does _this_ program store its data? Don't need to remember, data dir is mapped explicitly.
That said I haven't tried to use PostgreSQL myself directly, just tools that uses it like Gitea.
As someone else said, docker isn't about you, it's about everyone else. The extra complexity up front is so worth it for the rest of the team.
You bind a mount to where the data is onto the external system. That way only the important data is exposed. It's very clean, although it requires you understand docker a bit to know to do this. But for things like postgres and similar sprawling software that basically assume they're a cornerstone of your entire application and spread out as though this was the case, it's actually a very neat way of using them a bit without having them take over your machine.
This is something that can be useful for a developer too. Like my search engine software assumes it owns the hardware it runs on. It assumes you've set up specific hard drives for the index and the crawl data. But sometimes you just wanna fiddle with it in development, and then it can live in a pretend world inside docker where it owns the entire "machine", and in reality it's actually just bound to some dir ~/junk/search-data.
Although I guess an important point is that using docker requires you to understand the system better than not using docker, since you both need to understand the software you run inside the container, as well as docker. It's sometimes used as though you don't need to understand what the container does. This is a footgun that would make C++ blush.
Postgres is a poor example; it's far far easier to do `apt-get install postgresql` than to run it from a container.
The latter needs a container set up with correct ports, plus it needs to be pointed at the correct data directory before it is as usable as the former.
It was refreshing being able to say: this works perfectly from a base Ubuntu 20.04 container, launching this and that commands to install and run the software. Anything that diverges from there, you're on uncharted territory and/or is a problem in your system, not a defect in the software.
Instead of keeping some old machine with old dependencies because project is on life support and doing update to latest version of language/tools is unviable, you can just keep a docker container for it while machine it is running on is up to date.
It doesn't solve the problem of application, but it is no longer ops problem that app is old.
You can also do that partially, like keep old PHP version running with just fcgi socket exposd in a container, while rest of the app lives on "normal" VM (or other container, if that's what you want).
Granted, a lot of THAT benefits from the fact that the container actually has to be a documented flavor of good solid *NIX, unlike the "Windows Image X" which is usually just a big "whatever" of old iron, management fad spoor, and seven different kinds of antivirus software. Hell, last time I did this, IT couldn't even find me any kind of description of what the "official" windows image actually was. So no fault of Windows there, it's just that, as the big bus everyone rides in, it gets all the goop from everyone wrenching on it.
The last time I counted up all the different persistent, security-related agents running on my corporate Windows box, it was actually 14.
My theory is eventually things will come full circle and there will entire app ecosystems running in a unikernel executable which is running on physical hardware and the hardware itself is segmented. I can imagine companies like netflix and cloudflare having racks of servers with 1TB ram with no disk, booting a unikernel from the network for maximum throughput and latency and then everyone in tech follows along.
What people want is the ability to do all these things with a single command that they put inside a bash script.
Another solution would find a senior developer who can manage his dependencies properly, but I can't afford him. However, I can pay a junior ML developer wage and just throw what he develops in a docker image instead of wasting time and money on doing things the hard way for a prototype.
As engineers, we're supposed to be rational people, but unfortunately, we tend to forget that time and money are often happen to be the most important constraints we have to work around.
For toy/evaluation use, it's hard to beat "tweak compose file, docker-compose up". You now have a golden source of truth for how the entire application and all of its moving parts were set up.
Going back to deploying applications on bare metal feels positively medieval by comparison. From the admin side, configuration drift is effectively not a thing anymore. That's huge.
Provided all that, I don’t get why “devops” is even a high-paying job at most places except few really web scale ones. They are basically former sysadmins who reject anything except plugging colored squares into square sockets.
I can run that container in CI, in tests (with stuff like testcontainers), on my laptop, in production, on Kubernetes, in Mesos, Lambda, whatever.
Instrumentation usually ends up in Prometheus and Jaeger.
One company came close, they had someone dedicated to maintaining the dev tooling including the containers. It still didn't "just work" but they handled the troubleshooting, and the fix made it into the repo for the container so it wasn't just an unwritten adhoc fix I needed to remember. Close enough.
I totally get the concept, I wish it worked so seamlessly that the promise was realized. But since I usually work with tooling I am familiar with, there is no point for me personally. I can stand up a local env faster than I can troubleshoot a broken container. Since I am not interested in the infrastructure and just want to get to work, that is what I do.
[0] We don’t have concise terms for the types of distributed, complex systems which have become standard.
This is the most concise and exact definition I've heard lately.
My only problem with the consequences of this approach is that the amount of overhead for performing the same operations is astonishing.
Messages pass on a network instead of stsying in RAM, CPU context switches every other ms to handle IO for what could have been a simple function call, etc.
It's amazing when it's needed, but seeing this approach becoming the standard for a big part of the industry almost turns it into an environmental problem. How much energy is wasted juggling bits around?
However, I was specifically answering to the different case highlighted by the parent comment: how a potentially cohesive application is cut along some of its internal APIs, and some functionality is allocated to different processes living in different containers.
It is an extreme point in the continuum "single thread" -> "multi thread" -> "multiple processes" -> "fully distributed". In that continuum scalability increases, while efficiency progressively decreases.
Cornering oneself to a specific point in the design space is problematic, and for some cases has direct implications on how many resources are wasted.
A very didactic experience is, for example, running a simple local application under a microarchitecture profiler, such as Intel Vtune. It is not uncommon seeing that even straightforward C/C++ programs use a core resources less than 10%.
What I am reflecting about is that the choice of fragmenting that program among tens of systems (maybe in a scripting langiage) should be conscious, and done after encountering performance or scalability bottlenecks.
How much of the resulting total system workload would be useful work?
The quantity of potentially wasted resources is astonishing if you think about it.
That's just one example, though. It also lets you standardize the dev environment (work directly out of the build container directly, or use your host OS and run unit tests in a local build), and it allows for easy, standardized testing within the same environment.
How so? A container is the same architecture as the host, how does that help with cross compiling?
The power of docker compose for the development process is unparalleled. Being able to declare my entire local environment in a single file in a consistent way, create and destroy it with a simple command, and share it with my co-workers eliminates so many issues.
Then being able to package my application in way that is repeatable and predictable is great. Then that same artifact is run across multiple environments spanning data centers around the world and on other developers’ machines via their compose files if needed.
We test the crap out of all of this as well. Docker helps there to generate ephemeral environments to run the bulk of the integration tests against every time we push to git.
Its just so damn powerful compared to what we had before…
There's a lot of code I've written that's not deployable any more because the versions of the dependencies it was using don't exist on my OS any more, and are too old to install.
Also, from my misadventures with Ubuntu VMs I always ran into the problem of them breaking over time.
Exact reproducibility is nice for two scenarios: 1) academic research, and 2) very large-scale applications and deployments. For regular people writing boring small web apps, choosing a stable base image and pinning dependencies is good enough.
Consider also that your preferred programming language will also very likely not provide particularly reproducible package builds.
This just means you don't.
Try using `sudo apt install git=1:2.39.2-1ubuntu1`
That pins it to a particular version so that it should be reproducible.
I've never looked seriously into it, but my feeling is that distros will delete old versions as newer ones are uploaded: When I run "apt-cache policy git" in my Ubuntu, I only see a couple versions available to install, often other packages show only a single one (so, the latest).
However, much in the same way that if you actually take your build system seriously you'll store your application dependencies in a local proxy, you can run a mirror or proxy to hold these historical packages too.
Take a look at something like apt cacher, however it is a proxy cache so you can reproduce builds using the exact same package versions but if upstream delete old packages, and you want to roll back to one you haven't previously downloaded, then you are out of luck.
I was cobbling together build scripts for a mail system for Raspi/Armbian the last couple weeks. Very similar packaging stuff, but the number of little subtle differences in install/postinstall/prerm/postrm scripts took a generous level of spackling over to get just right.
Hell, once I get everything nailed down, I'm writing test frameworks for my bloody build scripts if you can believe it.
So...uh... Last week?
The freedom given in ability to limit your thinking space by reproducibility should not be under stated.
For almost all non-Google use cases, repeatable is generally good enough.
But then, we have none of those problems at work and people still believe it's a panacea. I don't know what people believe they gain by it.
In general, without some kind of qualification, deploying an image is way worse than a flat binary, or a tar from some interpreted language.
Yes you do need to configure some environment variables like in almost any other form of deployment.
Messing around with tar files or flat binaries doesn't work for me. I tried that. I build a python app into an executable with Nuitka but it generates a folder anyway. Ok throw it in a tar file. Nope, doesn't run because the glibc version is too new on the developer machine. Nice try. I have to build it on the target machine. Amazing.
I got stuck creating VMs for testers for a distributed workflow (several services, several tools), and keeping those VMs up to date and working was a PITA, but far less painful than dealing with them filing bug reports based on thinking they were doing X when they were doing Y.
I ended up creating a workflow that felt like layers, so when Docker came along I didn't need a salespitch. I could just do what I'd done with a script instead of a runbook. Where do I sign up?
Containers are packaging for executables.
MacOS makes static linking quite difficult.
And Windows is infamous for "DLL Hell."
Even windows reinvented the exe with msi's, and python has dists and wheels, and *nix/bsd has jails and vms have existed forever, and java has the jvm with jars and previously, applets.
Of course, none of those are actually as user friendly nor cross platform compatible as docker. Heck even docker isn't perfect and will leak abstractions once you start hitting syscalls (network, filesystem, etc), or if you're like 99.9...% of developers who don't host their own artifactory and cryptographically tag every step in their dockerfile
Show me an .exe that can self deploy an elk stack across windows, Darwin, and Linux with minimal system environment issues
Honestly, getting some desktop software in the form of one of those is wonderfully refreshing. Just take an AppImage file or whatever, make it executable and run it.
Sometimes you don't care about the benefits of shared libraries and want something to just run regardless of what's going on with your distro, for which these technologies are great. Ideally, in addition to the proper package manager approach for those that value different things at different times.
But for shipping things like WebApps, OCI containers and Docker/Podman is amazing.
There is a long, bloody history of packaging executables with only their direct dependencies.
Docker doesn't fix that. In fact, sometimes it makes it even harder for my testers to keep track of what is happening where. A shockingly high number aren't even able to visualize the logical boundaries between physically cohabiting, but logically partitioned systems.
It's too many damned layers.
And developers do awful things with docker containers to the point I end up having to rip apart multiple projects Dockerfiles to rebuild those, but with different values to work around the fact that oh .. look at that, this one spins up a postgres database that wants to bind to ports we're already using. Looks like I get to go dig into the base image to change that port and rebuild...
Again, not saying it doesn't work. As someone who put in the work to become a generalist though, I put in an asinine amount of time working around other peoples things that quote "make things easier".
Maybe for the person writing it. Certainly not for someone learning from it or having to analyze it.
The whole app can be started with a single command and it works on most Linux distributions. I can't imagine wasting the time of users or newbie developers demanding that they install all these things separately and with no easy way to clean it up if they want to undo everything.
After he spent 3 months building a web app, I asked him how he wanted to deploy it.
Perfectly straight face he said we would take his developer machine to a local data center and plug it in. We could then buy him a new developer machine. It went downhill from there.
I ended up writing the application from scratch and deploying it that same evening.
Owner hired a lot Of strange people.
I had one of these years ago where QA had an issue that I couldn't reproduce.
I walked over to his desk, watched the repro and realized that he was someone who clicked to open a dropdown and then clicked again to select, while I would hold the mouse button down and then let up to select.
I didn't know the latter was possible, but I also don't know which of these I do.
Dev would get the thing working on their machine configured for a customer. We'd take their machine and put it in the server room, and use it as the server for that customer. Dev would get a new machine.
Yes, I know it's stupid. But if it's stupid and it works, it isn't stupid.
DLL Hell was real. Spending days trying to get the exact combination of runtimes and DLL's that made the thing spring into life wasn't fun, especially with a customer waiting and management breathing down our necks. This became the easiest option. We started speccing dev machines with half an eye on "this might end up in the server room".
nm -D is not hard. Debugging missing symbols isn't even that hard.
Biggest damn challenge I see is we've completely dropped the ball at educating people about how linkers, loaders, and dynamic library symbols work.
...and so people understand something here... I've transplanted/frankensteined userspaces that were completely hosed/disjoint into working before. In fact, every now and again I do it just to remain in practice. It's actually gotten to the point I don't even worry about dependency hell anymore. I just find the right version of the library/source and expand my archives just in case I need it down the road.
Turns out they did ship a 1:1 image of his machine.
The OS, the development environment and the application (both code and live objects) where one and the same thing. To ship an "app" you would export the image and the user would load it into their Smalltalk VM.
And dependencies are about the way that we went from a blank slate to a working system.
If you can't retrace that path, you can't debug. If you don't have tools to track the path, you will make mistakes. At best, you can exactly replicate the system that you need to fix -- and fixing is changing.
Is there one command, or one script, which can do that? You need that, basically daily.
Is there one command to rebuild with the new libfoobar-1.17, and at most one more to test for basic functionality? You need that, too.
It's not like OS images are any better in this respect. I have bit my teeth long enough on software that depends on the base OS and not being able to infra upgrades. Bifurcating responsibility into app/platform is a breath of fresh air by comparison.
ldconfig -p | grep libfoobar
If you go and spread the linkers all over hither and yon and make it nigh impossible to get results out of them, or don't bother to keep track of where you put things... Welp. Can't help ya there. Shoulda been tracking that all along.
Oh, excuse me... That'll only work for dynamically built things in the abscence of statically linked stuff or clever LD_PRELOAD shenanigans.
Hope ya don't do that.
Fact is. You're really never going to get away from keeping track of all the moving pieces. You're just shuffling complexity and layers of abstraction around.
For example, it's incredibly common for Dockerfiles to contain commands like:
RUN apt-get update && apt-get install -y foobar
Which is to say, not in the slightest.
Every time you do something in a Dockerfile that involves pulling something from the network, you don't have reproducibility. Unless you pull something using a cryptographic hash (which I've never actually seen anyone do), the network can give you a different answer every time. The answer might not even be idempotent (calling it once changes the result of calling it again).
My argument was the same as yours. Virtually every dockerfile has an 'update' call as the second or third layer, which means none of the repeatable layers afterward (like useradd or mkdir) are actually repeatable. You're building a castle on the sand.
Docker images are reproducible. Docker builds are not. And that's okay, as long as everyone accepts that to be true. And if Docker contributors can't accept that then we are all truly fucked, until something else comes along that retreads the ideas.
The smallest example is:
pkgs.dockerTools.buildLayeredImage {
name = "hello";
contents = [ pkgs.hello ];
}1) you can apt-get a specific version
2) even if you don't apt-get a specific version, if two developers build and then run the container (within some time frame), they will get the same version. It would be rare to get a different one on the same day.
And 3) You can still apt-get a specific version
And 4) That's still far and a way greater chance two developers will have the same version when using docker in this way than if they're both using two crazy different versions of Ubuntu locally on their desktop.
Also most developers that are working on the same codebase will push to their branch and circleCI will make it so there's only one docker image, that's tested and moves along to prod eventually.
Proprietary software vendors and in-house corporate repos might pull sloppy crap here, though.
It is nice when your package manager reliably alerts you to changes like that, though! Binary package managers can't, really— although this feature could be added— but sourced-based ones often do. Nix and Guix enforce this categorically, including for upstream source tarballs for source builds. Even Homebrew includes hashes for its binaries, and provides options to enforce them (refusing to install packages whose recipes don't include hashes).
All of these "maybe it'll work" type ideas are what make modern software so brittle.
Well no, the layer won't be rebuilt if you already have it from yesterday or last week and that + previous lines in the Dockerfile didn't change. So yours would still be using the old version, but someone new to the project would get the most up-to-date version. Have to remember to skip the cache to get the most recent version.
When people say it needs to be reproducible, they don't mean "reproducible in the same host machine, with the same cache lying around".
- Docker Desktop and Docker engine (CE) behave differently, e.g. bind mounts, or file system ownerships.
- CPU/Platform differences (ARM vs. AMD64): many devs don't realize they use ARM on their mac, thus ARM images are used by default, and tools you run in it (or want to install) may be have differently, or may be missing entirely
- Incompatible Linux kernel APIs (when containerized binaries make syscalls not supported by the the host's kernel, for whatever reason)
- Using the same version tags, expecting the same result (--> insanity, as you know it :D)
- Different engines (e.g. Docker Desktop vs. colima) change the execution behavior (RUNNING containers)
- Different build engines (e.g. kaniko vs. BuildKit vs. buildah) change the BUILD behavior
For anyone who is interested: more details in [1].
[1] https://www.augmentedmind.de/2023/04/02/docker-portability-i...
A lot of people are in here complaining about how Docker is not reproducible enough. But reproducibility of image builds is a matter of diminishing returns, and there are other problems to worry about, like the ones you are pointing out.
Speaking of which, it's probably good to get in the habit of installing some Linux OS in a VM and trying to run your container images inside that (with "plain" Docker, no inner VM), before pushing it to your cloud host and waiting for it to fail there.
It's true that Dockerfiles are not reproducible, but at least they're human friendly and easy to deploy. If you need something more deterministic, I really encourage you to try NixOs. It's (almost) 100% reproducible and works for any real-world use-case that I've ever had. Dockerfiles have a different use case - they are a formal version of a installation instuction that you would give to a new hire in the older times.
Consider the following flake:
# flake.nix
{
outputs = { self, nixpkgs }: {
packages.x86_64-linux.default = with nixpkgs.legacyPackages.x86_64-linux; dockerTools.buildImage {
name = "nonreproducible-image";
copyToRoot = (runCommand "not-reproducible" {} ''
mkdir -p $out
head -c 100 /dev/urandom > $out/random
'');
};
};
}
It will produce a store path with the same hash on any machine you 'nix build' it on, but the contents of 'random' will be different each time.
Flakes do not make nix reproducible since flakes do not prevent derivations from running arbitrary commands and producing their own impurities.
Instead, Nix ensures the inputs are fetched deterministically via Fixed Output Derivations (FODs) that guarantee the output of building/compiling against that input can only vary so much, and in ways often insignificant to code flow in the output program, (timestamps/byte layout, etc). In the Nix thesis this is referred to as "the extensional model" in Section 5 (Page 87), otherwise known as input-addressing rather than the intensional model which is in reference to content-addressing.
Of course we'd all like build outputs to be deterministic, but that's not very pragmatic, or achievable. Many compilers and toolchains don't produce the same results twice, but this often does not effect the behavior of the program. There's some tracking of that goal on r13y.com, and it's probably a goal that will always be limited to a small subset of deterministic outputs, as it's unlikely that all toolchains used in the dependency graph will exhibit deterministic behavior.
Sometimes things that do effect code flow leak in, but are squashed by the Nix stdenv where possible, and also by the culture of fixing/patching those problems in Nixpkgs. The result is a pragmatic set of 100,000+ packages that work properly and can be reproduced anywhere.
1) One should neither be using the "latest" nor just the "version" tag as the version can still vary depending on when it is pulled.
Instead, one should use a combination of version + hash, say alpine:3.18.2@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 for reproducibility reasons. This provides for human readable versions as well as the specific hash.
2) Next, afaik, Compose has removed the need for version tags. All of the compose.yml files that I now use do not specify versions.
See https://github.com/compose-spec/compose-spec/blob/master/04-...
That's how we get hacks like SolarWinds and MOVEit.
"It works on my machine"
"Okay well here is the exact image digest and configuration from docker inspect"
"Thanks I can reproduce the problem now"
People want and need the ability to pin their environment. If you want to avoid silly surprises when going to production or through your CI system, you need everything to be identical: same versions, same configuration. Tooling should help you do that not introduce new way to shoot you in the foot.
docker shouldn't be used for everything. if you provide a docker version of something, it's a smart idea to also publish (for example) an appimage or deb file for people who can't or don't want to use docker.
like for example, at my work we don't want to use docker because we will have to get approval from corporate for every little script we want to run in a container because corporate identifies a container as a separate application so it must go through the approval process (which takes 4-8 weeks).
Containers are still a great improvement over running on dev machines directly: besides cleanly separating environment per project, an unsung benefit is being able to get rid of the environment when you're done working on a project instead of accumulating bloat over time.
Somehow the old adage of portable software being good software seems to have been lost to the ages now that we as developers have attained such precise control over the environment our software runs in.
FROM scratch
COPY this-or-that
LABEL prometheus.port=9100
LABEL prometheus.path=/metrics
EXPOSE 3001
EXPOSE 9100
Only feeble component is CA bundle for SSL-related stuff as that by nature is changeable.
In my case, the bottom bid contracting firm that delivered the code had special logic for windows but otherwise wouldn’t happen on unix based machines.
Sure, but containers don't protect you from that - you're still exposed to differences in the host kernel. For that kind of issue you'd want a full VM rather than a container.
You've gone from being exposed to OS and kernel differences to being exposed only to kernel differences. For most apps, that's acceptable. Others will need to ship VMs. Some will even need to ship bare metal.
It's a tool, not a dogma.
> It's a tool, not a dogma.
The idea that containers are the only or right way to do service orchestration is absolutely dogma, enforced by kubernetes and friends.
I dunno, probably stuff like ulimits and selinux rules that you don't think about until it burns you. Not to mention whatever idiosyncratic configuration your customer might perform.
> [It's] absolutely dogma, enforced by kubernetes and friends.
The dogmatic people are wrong to be dogmatic, but in the context of this particular conversation, I would like to point out - meaning no disrespect - that it is you who is advocating the more maximalist and less pragmatic view.
How so? The thread started with talking about having a statically linked binary (not a specific one with special requirements, but a generic one), and described putting it in a container with a config to expose some ports from it. That seems very much like a dogmatic everything-must-be-a-container position. I'm not arguing for never using containers, I'm arguing for using them where they make sense and not using them where they don't.
If I'm misreading you, then I apologize.
- Check for processes called "sleep", exit if such process exist (or try to kill it).
- Reset $PATH to default value then expect to find a _very_ specific ancient version of system utilities there.
- Create files in /tmp with fixed names. Fail if they already exist. Forget to delete them on failure.
- Walk entire filesystem searching for something. Crash on broken symlinks.
- Enumerate network interfaces. Crash if more than 7 are present.
- Hardcode _both_ specific user name and associated UID.
- Put temp files all over place, including into application directory
- Ignore $HOME and use homedir from /etc/passwd, then create lock/config file under homedir. And you want to run two instance of this app in parallel.
But the code I nicked the example from was actually our internal k8s/docker deployment testing/debug app, so, well, it's in containers by design.
No, containers are not a problem and they don't add any additional problems.
It's not more complex by any stretch
In retrospect this is not completely surprising given the incentive system in science.
Also, what kind of scientist was this? I'm a physicist. I'm deeply concerned about the reproducibility of my results. I periodically try to rebuild my software systems on a clean computer to make sure it's both possible and well documented.
By way of comparison, biology and related 'sciences' ( psychology, medicine, nutrition, microbiology etc.) will appear to be nearly a scam...
My wife's attitude is typical: If they want to touch my computer, they can buy one that they own.
In this case the scientist was a postdoc so they partly used the group lead's funding and partly had their own funding. That money was for science, not buying computers. It paid for NMR machine (a $10M device), solvents, reagents, going to conferences, grad students, color figures in papers... but my professor was fairly well-funded so we had a full-time "professional" sysadmin (not a grad student) and everybody had a $15K SGI Indy on their desk (which was abysmally slow, because it was the lowest of the low end capable of running OpenGL).
Otherwise distro package installs are not reproducible even if you lock the base image (and apt-get with a specific version will most likely fail).
alternatively:
docker buildx build --platform linux/arm64,linux/amd64 # or whatever you need to target...
Good for one off testing but not a full solution
Nix.
For those focusing on the title, a nice way of seeing a difference is less "my container does X" and more "when I run the production container...". Sure your local build might do one thing but you can get what's built in production and try running it. Maybe you can quickly swap between multiple combinations of service versions to replicate a bug that happened during deployments.
Despite these hot takes containers are a massive step in simplifying system deployments.
dig +short dwdraju.medium.com
162.159.152.4
162.159.153.4one thing containers do is provide an audit/reproducible basis to say WHY something builds. it may still be an implicit dependency: something in a specific version of an underlying substrate like the OS or a boot time act which works on that container and not on another. You may not "see" it at first but its a lower bar to understanding it, if the revision control behind the build chain is good enough.
a build chain of :latest is probably not good. reproducible builds drive against that.
I don't build things for a living any more. Probably this is all well known and argued better by others.
That presentation is a borderline-psychotic rant unless viewed through a very narrow lens where the only thing that matters is maximum-performance systems programming.
He's disregarding every productivity improvement in pursuit of maximum performance. He makes that very clear, and in that context it makes sense, but most people won't have the same priorities because we don't live in a world where we can afford to write our own TCP/IP stack to maximize the throughput on our shitty REST APIs that C/R/U/D our customers' TODO items in our database.
And it's true that not everyone has to run their app at 120hz latency, but it's also true that software has become slower.
And it's not something new.
JWZ from Netscape established the Zawinski's Law:
https://softwareengineering.stackexchange.com/questions/1502...
>> Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.
But also N Wirth said something similar:
https://en.m.wikipedia.org/wiki/Wirth%27s_law
>> Wirth's law is an adage on computer performance which states that software is getting slower more rapidly than hardware is becoming faster.
Wikipedia from Wirth Law surveys other variations of the statement.
So no, it's not just some crack pot hypothesis. It's a real world problem that we have to actually deal with.
Nix plus good UX that can even build vscode compatible devcontainers that mirror the nix shell.
Recently changed a codebase from a devcontainer to Nix shell:
Test runtime docker: 13s
Test runtime nix: 6s
:shakes-fist-at-bazel-for-not-being-easier-to-use:
I think that using less software, and better software is part of the solution to making reproducibility easy. But FWIW, I do think that Nix lowers the barrier to making reproducible software.
L O L