Hacker News Reader

Comment by contingencies | Hacker News Reader

Ouch. This is the security posture that half a billion dollars, hundreds of engineers, multiple layers of abstraction to a firewall ruleset and each Stream-aligned team with an embedded Product Manager and UX Designer[0] buys you. I too contributed security fixes to Docker in the early stages but soon gave up.

Docker is a fascinating study in hype. Containers already existed, docker wrapped them very badly, raised half a billion dollars, popularized their wrapper (I understand kubernetes is the fad-wrapper now) and then failed to obtain a revenue stream. I didn't see them return much to open source, though perhaps they contributed to projects I don't see.

Honestly, the whole thing has a plotline like a Shakespearean tragedy. Someone could do a film, starting with interviews with the guys who wrote the original Linux containers and namespacing kernel code at IBM, the maintainers of jails in FreeBSD, and an early hacker at VMWare.

[0] https://www.docker.com/blog/building-stronger-happier-engine...

View on HN

The Docker UX is what made the difference, not the underlying tech. Building containers in an easy incremental and reproducible way that could then be stored in a repo and pulled over HTTP was the evolution in packaging that was needed.

UX alone can be a multi-billion dollar business, for example Dropbox was just a folder that syncs. I'll leave you to find that classic HN comment about it. Docker just executed poorly and didn't have much worth paying for.

How does one build containers reproducably? I can make parts of my stack reproducable if I use existing, pinned containers, but due to all the `apt-get`ing in my dockerfiles, I'd call them anything but reproducable.

I've seen success building them in centralized server(s) and 'apt-get'ing specific dependencies instead of the latest ones.

With apt2ostree[1] we use lockfiles to allow us to version control the exact versions that were used to build a container. This makes updating the versions explicit and controlled, and building the containers functionally reproducible - albeit not byte-for-byte reproducible.

[1]: https://github.com/stb-tester/apt2ostree#lockfiles

that's where docker failed in only copying part of what I built (they simplified it, and to an extent that was good for making a product, but I had proposed the need for (and prototyped) a container/layer aware.

see https://www.usenix.org/conference/lisa11/improving-virtual-a....

the "problem" is building that layer aware linux distribution, so Docker just skipped it and relied on existing things and building each image up linearly.

This was actually an advantage of CoreOS's rkt (and the ACI format), where it enabled images to be constructed from "random" layers instead of linear layers (though the primary tooling didn't really provide it and were just docker analogues, there were other tools that enabled you to manually select which layer IDs should be part of an image - but docker "won" and we were left with its linearized layer model).

The Docker images are not even versioned.

You use Docker as a distribution mechanism for the output of a reproducibility-oriented build system like Guix or Nix. Pin your dependencies inside the build system, then the build system can generate a container.

I use NixOS. But I see no reason to dabble with containers since I use NixOS except for when I can't coax some software to work reliably on NixOS or I want to produce binaries that work on distros that are not NixOS, in which case I can't use NixOS based containers anyway.

> I can't use NixOS based containers anyway

I'm no expert, but I believe the purpose of containers is to include all user space dependencies... so this doesn't make sense to me.

I personally am surprised NixOS hasn't leaned into marketing itself as ideal for creating reproducible containers.

Its great when it works, but I use containers to emulate normal Linux when it doesn't.

If you're using NixOS on the desktop, sometimes you want to run an obscure piece of software quickly, without packaging it for NixOS, which can be a PITA with proprietary software or software with irregular build processes.

Or maybe you're working in an organization where some instructions are written for another distro, and you just want to be able to follow them word for word the first time you attempt a task, to make sure you understand the process on a 'normal' distro, or because you're troubleshooting with someone who is running another distro. Then NixOS' support for running containers is handy, but afterward you're left running some containers whose reproducibility doesn't match what you've come to expect frok the rest of your system.

> I personally am surprised NixOS hasn't leaned into marketing itself as ideal for creating reproducible containers.

Agreed, I think this is a really good use case for a lot of companies.

I had in mind that your goal with reproducible containers was to distribute some app your business makes on platforms that don't know anything about NixOS, in which case building the app with Nix and then using your favorite container runtime as the deployment target makes sense. But if you're using containers as an escape hatch for NixOS and Nixpkgs, that definitely doesn't do much for you!

In enterprise environments, some old school distros have an archiving layer that sits between hosts and their normal repos that you can use to hold back updates. Maybe you could use something like that. I forget what Red Hat's offering is called but I think it's part of Satellite. Idk if there are any free tools for that, but maybe there are.

The other alternative escape hatched that NixOS has, like FHSUserEnvs or just steam-run, you likely already know about.

But that's the good thing about them. It isn't just a VM snapshot it is a recipe to build an image.

Pre container times there were recipes to build VMs including plain shell ( which can also be used to build containers).

Docker, the FOSs tools and the hub are very nice packaging.

Sure, but those were, and still are, a nightmare to use/learn for a lot of usecases.

How reproducible is up to you.

You can build everything elsewhere and only copy in the artifacts. Or only install pinned versions of things. Or only use certain base tags. Or do the entire CI thing inside a container. Or use multi-stage containers for different steps.

My previous startups all used pinned based images with multi-stage containers to build each language/framework part in its own step, then assembly all the artifacts into a single deployable unit.

I use Nix, which has a `dockerTools.streamLayeredImage` function: https://nixos.org/manual/nixpkgs/stable/#sec-pkgs-dockerTool...

> but due to all the `apt-get`ing in my dockerfiles, I'd call them anything but reproducable.

Giving this some thought, I agree if you use package managers and base images I don't think it's possible to build images in a perfectly reproducible fashion.

But I would ask: don't image tagging and registries to provide a reproducible output (binary, packages, etc.) make it unnecessary to provide reproducible builds? Why does it matter if the build is perfectly reproducible if the actual output of the image can be reproduced? I.E. if I can run the software from an image built two years ago, does it matter if I can't build the exact Dockerfile from two years ago?

EDIT: fixed some grammar issues

I don't know, in my opinion then you are just running a container, like running an exe. You really aren't reproducing anything.

A lot of the value of containers seems to be in providing the ease of deployment benefits of a static executable to languages that don’t support compiling to a static executable.

Isn't that the whole point of a container is to not retain state of anything else within the filesystem other than the desired entrypoint/cmd(s)?

It's all relative, and as long as you don't use rolling-release distro, docker is a good compromise between having security fixes and partial reproducibility where it matters.

Remember, there were lots of containers before docker: jails, schroot, lxc, etc.. But the setup model there was "do the base os install, then enter the container and run setup commands" -- which is as non-reproducible as it gets, those commands are often not even saved anywhere. (Yes, you could if you are diligent about it. I don't think many people did that, certainly no tutorials mentioned it)

Compared to that, Dockerfile was revolutionary: a _requirement_ that all software is installed via automated means, and no simple way to do "just one more adjustment" in manual, undocumented way.

And then there was rkt around the same time as Docker and thought they would be able to take the mantle when Docker-the-company struggled to make money. The biggest problem with that theory is that it had none of this secret sauce you mention.

When I would point this out I usually got brigaded. Now RedHat owns the company that made rkt and I don't know if it's even being developed anymore. But nobody brings it up so I don't even have to think about it.

rkt (and many other container solutions) was introduced after docker was released and became popular... they even mentioned docker's shortcomings as a motivation for the project creation [0]. It had all the same problems as other replacement software: there were plenty of bugs and missing features (the announcement mentions "prototype quality release"), documentation was limited, and there are no community to help you. None of those would be fatal if it was significantly better than docker, but it was not -- it had less functionality and needed more scaffolding. So almost no one made the switch. It is closed now [1]

And why "rkt"? There were much better alternative container runtimes. For example Sylabs Singularity [2] -- container-as-a-file, instant mounting, etc... I wish more people knew about it.

[0] https://web.archive.org/web/20141201181834/https://coreos.co...

[1] https://github.com/rkt/rkt#warning-end-of-project-warning

[2] https://github.com/sylabs/singularity#singularityce

I don't think it was even that. It was massive marketing and hype which drove docker adoption. The engineering concerns and the user experience were both negatives because they actually added a considerable cost and friction to the experience.

I will obviously need to back that comment up but fundamentally the security posture of docker (and most package repositories) is scary and the ability to debug containers and manage them adds considerable complexity. Not to mention the half baked me-too offerings that circle around the product (windows containers anyone?)

The repeatability argument is dead when you have no idea what or who the hell your base layers came from.

What it did was build a new market for ancillary products which is why the marketing was accelerated. It made money. Quickly.

I mean, these are the same people who released an entire new daemon in the 2013 that ... can only run as root. Party like it's 1983. I believe they added some experimental caveated "non-root" mode last year, but how this was not even supported, much less the default, from day one is just ... ugh.

That being said, Docker is more than just "Linux containers" or FreeBSD jails; the key thing it brought was the push/pull commands, which made setting up and distributing containers a lot easier, as well as some other UI/UX things. I think these ideas are implemented badly, but they do add value.

Compare this with FreeBSD jails, where the instructions were "download the base set from the FreeBSD ftp and extract it to the destination dir, manually set up /dev, twiddle with /etc/rc.conf, muck about with ifconfig for networking, etc."

Looking at the current handbook[1], things are better now: there's bsdinstall and /etc/jail.conf, but e.g. jail.conf was added only a few months before Docker was released, but the key value proposition of Docker (push/pull) is still missing. Right now, setting up nginx or PostgreSQL in a FreeBSD jail is more work than setting it up system-wide. People don't use Docker because it runs in containers per se, they use it so they can run something in one easy step.

IMHO Docker is more or less the Sendmail of our age[2]; we just have to wait for "Docker Postfix" or "Docker qmail" :-) I'd make a punt, but I don't really have that much time.

[1]: https://docs.freebsd.org/en/books/handbook/jails/#jails-buil...

[2]: People have forgotten now, but Sendmail solved a lot of issues in the early 80s and dealt with the wildgrowth of different email systems better than anything else; while crusty and ugly, it added a lot of value when it was first released.

Wouldn’t "Docker Postfix" or "Docker qmail" be podman?

Maybe; I haven't used it. Whenever I needed containers I used runc or just set up some cgroups manually, but it's been a while since I had to do so.

I'd rather run Docker as root than enable unprivileged user namespaces, security-wise.

Why, exactly?

Personally I would trust the Linux kernel developers to find and fix security issues around unprivileged user namespaces much more than I trust the Docker team to build a secure product.

> Personally I would trust the Linux kernel developers to find and fix security issues

Yeah I highly recommend not having that view. Kernel upstream is the entire reason this problem exists - they spent decades downplaying and deriding security researchers who found root -> kernel privesc, and, in general, have had an incredibly hostile relationship with security professionals.

I don't know the case with Docker as much but my impression is a lot more positive based on what I've seen - integration with Apparmor/SELinux, seccomp, memory safety, etc.

Docker non-root is some kind of sad twisted joke. The person that jumps through pages of hoops to get that mess working and the person that reaches for Docker in the first place are not the same person. When they say they recommend Ubuntu only, they aren't fucking around. I tried last year on Debian. Never again. I nuked it all and started over with regular root Docker and restored whatever was left of my sanity.

Docker compose is also a bewildering mess of conflicting versions and baffling decisions. With incredibly poor documentation. Try to figure out network_mode without spending hours doing trial-and-error github issues stackoverflow head pounding.

I think the UX is fine if you ignore Dockerfile and docker-compose.yml. But those files are rather atrocious. The Faustian bargain of Docker is you fetch images from rather dubious sources and of dubious origin, run them with a root daemon, and let Docker molest your iptables. In return, you get the illusion of having accomplished some form of meaningful isolation. No security whatsoever. But hey, I get to run two versions of node on one Linux and pretend I didn't just sweep all these serious issues under my, now, rather large bed.

> Try to figure out network_mode

Try to figure out why bridge isn't a bridge (honestly it still makes my blood a couple degrees warmer every time I remember that)

I mean, these are the same people who released an entire new daemon in the 2013 that ... can only run as root. Party like it's 1983.

Small correction: that would be 1982. Bill Joy released chroot in 1982.

Docker accidentally(?) created my very favorite cross-distro Linux server package & service manager.

That it uses containers is just an implementation detail, to me.

All the docker replacements will be useless to me if they don't come with a huge repository of pre-built relatively-trustworthy up-to-date containers for tons of services.

I love no longer needing to care much about the underlying distro. LTS? Fine. I still get up-to-date services. Red hat? Debian-based? Something else? Barely matters, once Docker is installed.

On my home server it even saves me from ever having to think about the distro's init system.

Bonus: dockerized services are often much easier to configure than their distro-repo counterparts, and it's usually easier to track down everything they write that you need to back up, and that's all the same no matter what distro they're running on.

>huge repository of pre-built relatively-trustworthy >relatively-trustworthy

yeah, that's not really the case with docker either I'm afraid

> yeah, that's not really the case with docker either I'm afraid

I feel like two places I disagree with this statement:

- Docker's official image library

- When a project you are looking to run has an official image they maintain

When those constraints are not met, I try to build my own image (based on the official images).

I try not to trust any other external images (with some exceptions from reputable sources such as linuxserver.io or Bitnami).

Shrug just about every service I want to run has an official image. I use a couple unofficial ones (e.g. Minecraft, and I think my Samba container—oh man, talk about being easier to configure than typical distro packaging—is unofficial) for my personal use and it's pretty easy to read the Dockerfile and build them myself, so that's not a big deal. At work, most any database or self-hosted service or anything like that has an official image.

So yes, it is.

> Bonus: dockerized services are often much easier to configure than their distro-repo counterparts

A hard no there.

Sure, some projects, after they started to publish their own Docker images concluded what it would be beneficial to expose more configuration options not buried in tons of .conf files - ie they started to support environment variables or have some script in the init what would dump the variables to the proper places.

Sadly, this works only for some projects, because if this is is an afterthought then you will have a pretty bad time figuring out why something doesn't work or not configured as you wish despite having all the knobs specified. The worst offender for was probably Gitea - it just ignored half of the settings until I found a combination what would trigger supporting some settings (!). Another offender is probably Nextcloud, it is easier to docker build a full image than to add SMB support to their official one with a published instructions on how to do so. And half the time it ignores 'trusted_hostnames' so it fires up and doesn't allow you to login because it doesn't know what 'nextcloud.yourdomain.tld' is it's own address.

Oh, don't let me start on Swarm on compose file shenanigans.

Just forcing them to record exactly where important files are saved and config files live has been hugely helpful. I like that it makes it easy to keep my config from being shit all over /etc with a thousand other files I will never want or need to touch, so my modifications to /etc on a typical server are small and tightly confined to a small number of files. What do I back up? It's right there in the "mounts" list. Have I backed up everything important? Add some dummy data, remove the image and do an "up" and if it still looks right, then yes, I have.

... and I don't have to re-learn any distro-specific config or packaging, for those dockerized services. Get it right once, and you're done.

> The worst offender for was probably Gitea - it just ignored half of the settings until I found a combination what would trigger supporting some settings (!)

Haha, that's actually one I run, and also one of the worst I've ever encountered. I found the Rootless version of their image much easier to deal with (I never got the other one to work at all) personally, for whatever reason. Part of the trouble is that Gitea's config file, and documentation thereof, is kinda a damn mess.

> my config from being shit all over /etc with a thousand other files

Yea, I was so involved in the previous comment what I forgot to say what this is what is actually helpful

> What do I back up? It's right there in the "mounts" list

Yep, I completely ignore the docker persistent volumes. My infra is running in VMs which are backed up by Veeam, so there is a bazillion of ways to recover data and services - and no need for full restores and volumes juggling if I only need a one file.

I get that this comment is currently being pummeled from all sides, but just to be clear on what Docker offered, and the value I think they created, it wasn't containers, but:

* The public registry. Obviously, there are a lot of downsides to this with respect to security, but the convenience of "just install docker and then you can run basically anything that has already been uploaded to Docker Hub" can't really be overstated.

* OCI. They were kind of forced into this much later on, but it's great to have a standard for how to package a container image and a bundle that can be loaded into a runtime. Containers may have already existed, but there was not standard way to package, deliver, and run them.

* The Dockerfile. For better or worse, it isn't perfect and is full of footguns, but as the "hello world" equivalent of build systems go, it's pretty easy to get started and understand what it's doing.

* I think, but am not certain, that Docker was the first to combine containers with union filesystems, saving quite a bit of disk space when you're running many containers that share layers. This also helped usher in the era of read-only root filesystems and immutable infrastructure.

It's also important to remember that developers are not the only people involved in delivering and running software. I see a lot of complaining here that containers add friction to their workflow, but look at it from the perspective of systems administrators, IT operators, and SREs that have to install, configure, run, monitor, and troubleshoot potentially hundreds of applications, some internally developed, some third-party, all using whatever language, whatever runtime, and whatever dependency chain they feel like without regard to what the others are doing. Having a standardized way to package, deliver, and run applications that is totally agnostic to anything other than "you need Linux" was pretty powerful.

Docker's hype train was absolutely absurd, but it was instrumental in making containers ubiquitous, and the wrapper is the reason why. Containers were painful as hell to set up and use; it was like using Git, but worse. Docker made it convenient, simple, and useful. By now everybody understands that immutable artifacts and simple configuration formats are extremely effective. Docker is one of the only tools I can think of that was designed with a human being in mind rather than a machine.

Docker is like brew in that they are both huge productivity levers, except Docker found a way to pay a bunch of technologists with VC dollars to evangelize the model.

Security is an afterthought in docker, deploy first questions later the dirty harry way. I learned lots about dockers lack of defense while trying to harden my instances and questioned why some of the options weren't enabled by default (--cap-drop=all and --security-opt=no-new-privileges and userns remap) other than exposing bad dockerfiles practices.

I’m not sure that is entirely fair.

Sure containers like LXC and bsd-jails existed, but I think the feature that won the hearts of devs was the incremental builds that let people fail quickly in a forgiving way.

With jails, the best I know of achieving this is creating a base jail filesystem (thin jail), and creating a new one. (Apologies in advance if another tool addresses this).

HP-UX vaults date back to late 90's, as one of the first container models for UNIX systems.

1. Docker has contributed a lot and provided if not technology at least ideas others have iterated on.

2. k8s is not a fad. Nor is it wrapper or really even equivalent with docker the container bit.

Docker is an imperfect solution to problems we shouldn't have after decades of Linux and Unix and its value is exactly in its DX.

But considering the pride in finding the right incantation to build C/C++ code, to set up a system service, to set up routing rules or any other advanced feature... I guess anything that tries to hide it will count as hype

> Containers already existed

Fascinating. I'm admittedly very new to container-land, but my understanding was precisely the opposite - that Docker invented this concept and practice of building a runtime environment that can then be replicated and deployed to many hosts. Do you have any search terms or articles I should pursue to learn more about the predecessors?

I've read through this[0], but it mostly suggests that approaches prior to Docker provided isolation/jailing, which is an important part of containerization but neglects a lot of tooling around building, deploying, etc.

[0] https://blog.aquasec.com/a-brief-history-of-containers-from-...

EDIT: various other comments ("The Docker UX is what made the difference, not the underlying tech. Building containers in an easy incremental and reproducible way that could then be stored in a repo and pulled over HTTP was the evolution in packaging that was needed.", "I think the feature that won the hearts of devs was the incremental builds that let people fail quickly in a forgiving way", "Containers were painful as hell to set up and use; it was like using Git, but worse. Docker made it convenient, simple, and useful. By now everybody understands that immutable artifacts and simple configuration formats are extremely effective") seem to support this perception.

EDIT2: Thanks to all those that replied!

Containers absolutely existed before. THere was a long-running project, I think it was LXC but there may have been one before that, which was doing great work fixing all the underlying problems before the technology went exponential.

> I think it was LXC

Still is :)

https://en.wikipedia.org/wiki/LXC

I belive that Solaris (OpenSolaris) Zones predates LXC by around 3 years. Even when working with k8s and docker every day, I still find what OpenSolaris had in 2009 superior. Crossbow and zfs tied it all together so neatly. What OpenSolaris could have been in another world. :D

OpenVZ is around since 2005, see https://en.m.wikipedia.org/wiki/OpenVZ and it had checkpointing, live migration between hosts, dump container to tar, etc before Docker even existed.

And linux vserver* has been around since 2001.

If you were willing to use out of tree patches, linux containers pre-date Solaris containers by years (Debian distributed pre-patched kernels pretty early on, so you didn't need to deal with compiling your own kernels, and Debian also had an option for vserver + grsec patched kernels). But, freebsd jails beats linux vserver by a year.

* not to be confused with the similarly named linux virtual server (LVS) load balancer.

HP-UX Vaults were introduced in late 90's, then came Solaris Zones, BSD jails,...

This on the UNIX world, mainframes had a couple of decades earlier.

Containers at their very heart in Linux are three ingredients: cgroups, namespaces, and filesystem-level seperation.

Here's an interesting way to learn more about this - Bocker, an implementation of docker in 119 lines of bash:

https://github.com/p8952/bocker/blob/master/bocker

> failed to obtain a revenue stream

Docker is throwing off incredible amounts of cash with Docker Desktop licensing alone.

Sadly. This is why I recommend running docker inside of a container (e.g. LXC) or a VM, as I just don't trust the security profiles for docker images.

so, uhm, before Docker, what was the one-liner that allowed me to get a ubuntu, fedora, nixos, ... shell automatically ?

You can do something with lxc - I had to use it instead of Docker on a Chromebook a while back. The wrapper isn't as nice, but it's not actually that hard to get going.

tar -zf ubuntu.tar.gz && chroot ubuntu

ish

    $ tar -xzf ubuntu.tar.gz && chroot ubuntu
    tar (child): ubuntu.tar.gz: Cannot open: No such file or directory
    tar (child): Error is not recoverable: exiting now
    tar: Child returned status 2

if I have to do wget of something (and find the url to wget) it's not worth it

I would use debootstrap.

https://semjonov.de/posts/2021-09/minimal-ubuntu-installatio...

If it requires to read a whole blog article like this to setup someone can make and sell a product with a one-liner command that does it. They could call it, idk, decker or something

I take it you don't remember learning Docker? I had to read way more than one article to fully understand it. I also had to learn what cgroups were and all kinds of things.

    tar -zf ubuntu.tar.gz
    tar: Must specify one of -c, -r, -t, -u, -x

Doesn't work on my shiny macbook. I'll stick with Docker, I guess.

lxc-create -t ubuntu

How is K8S a fad?

Ask me in 10 years, I'll tell you whether it was a fad or not.

a 20 year run in tech is not a "fad".

"tech" was a fad to your future AI overlords

On a long enough timeline, everything can be considered a fad!

The shortening of the name to what you have above is your first clue.

Uh-oh, guess we have to tell everyone internationalization is just a fad because everyone writes it i18n.

It absolutely is. Approximately no-one gets real business value out of internationalization, they just use it as a stick to beat technologies they dislike.

This is hilarious.

It's not a fad, but a fashion. People wear it because it looks cool on your CV or consultancy offering, not because it keeps you warm and dry. They don't even understand it properly. Even AWS, who I work with regularly on EKS issues, seem not to understand their own offering properly.

Ultimately in a lot of companies it ends up an expensive piss soaked nightmare wearing a jacket with "success" printed across it.

I like the idea but the implementation sucks and echoing the root poster, containers have some unique problems which can indeed cause you some real trouble and friction. It makes sense for really horizontal simple things, which isn't a lot of stuff on the market.

I used borg for over a decade, I know what k8s is for.

Reply on news.ycombinator.com