How does one build containers reproducably? I can make parts of my stack reproducable if I use existing, pinned containers, but due to all the `apt-get`ing in my dockerfiles, I'd call them anything but reproducable.
Remember, there were lots of containers before docker: jails, schroot, lxc, etc.. But the setup model there was "do the base os install, then enter the container and run setup commands" -- which is as non-reproducible as it gets, those commands are often not even saved anywhere. (Yes, you could if you are diligent about it. I don't think many people did that, certainly no tutorials mentioned it)
Compared to that, Dockerfile was revolutionary: a _requirement_ that all software is installed via automated means, and no simple way to do "just one more adjustment" in manual, undocumented way.
When I would point this out I usually got brigaded. Now RedHat owns the company that made rkt and I don't know if it's even being developed anymore. But nobody brings it up so I don't even have to think about it.
And why "rkt"? There were much better alternative container runtimes. For example Sylabs Singularity [2] -- container-as-a-file, instant mounting, etc... I wish more people knew about it.
[0] https://web.archive.org/web/20141201181834/https://coreos.co...
[1] https://github.com/rkt/rkt#warning-end-of-project-warning
Giving this some thought, I agree if you use package managers and base images I don't think it's possible to build images in a perfectly reproducible fashion.
But I would ask: don't image tagging and registries to provide a reproducible output (binary, packages, etc.) make it unnecessary to provide reproducible builds? Why does it matter if the build is perfectly reproducible if the actual output of the image can be reproduced? I.E. if I can run the software from an image built two years ago, does it matter if I can't build the exact Dockerfile from two years ago?
EDIT: fixed some grammar issues
Docker, the FOSs tools and the hub are very nice packaging.
see https://www.usenix.org/conference/lisa11/improving-virtual-a....
the "problem" is building that layer aware linux distribution, so Docker just skipped it and relied on existing things and building each image up linearly.
This was actually an advantage of CoreOS's rkt (and the ACI format), where it enabled images to be constructed from "random" layers instead of linear layers (though the primary tooling didn't really provide it and were just docker analogues, there were other tools that enabled you to manually select which layer IDs should be part of an image - but docker "won" and we were left with its linearized layer model).
In enterprise environments, some old school distros have an archiving layer that sits between hosts and their normal repos that you can use to hold back updates. Maybe you could use something like that. I forget what Red Hat's offering is called but I think it's part of Satellite. Idk if there are any free tools for that, but maybe there are.
The other alternative escape hatched that NixOS has, like FHSUserEnvs or just steam-run, you likely already know about.
I'm no expert, but I believe the purpose of containers is to include all user space dependencies... so this doesn't make sense to me.
I personally am surprised NixOS hasn't leaned into marketing itself as ideal for creating reproducible containers.
Or maybe you're working in an organization where some instructions are written for another distro, and you just want to be able to follow them word for word the first time you attempt a task, to make sure you understand the process on a 'normal' distro, or because you're troubleshooting with someone who is running another distro. Then NixOS' support for running containers is handy, but afterward you're left running some containers whose reproducibility doesn't match what you've come to expect frok the rest of your system.
> I personally am surprised NixOS hasn't leaned into marketing itself as ideal for creating reproducible containers.
Agreed, I think this is a really good use case for a lot of companies.
You can build everything elsewhere and only copy in the artifacts. Or only install pinned versions of things. Or only use certain base tags. Or do the entire CI thing inside a container. Or use multi-stage containers for different steps.
My previous startups all used pinned based images with multi-stage containers to build each language/framework part in its own step, then assembly all the artifacts into a single deployable unit.