Non-Obvious Docker Uses | Hacker News Reader

Non-Obvious Docker Uses

matt-rickard.com

210 pointszdw93 comments

We actually use docker multistage build as mentioned here as a replacement for make file. The nice thing is you can run it locally to build but also run it in a podman container as a part of CI/CD process as is without any changes.

Nice thing about this strategy is that you can uniformly enforce dependencies, versions, etc

The horrible thing is that it is slow, hard to debug and develop on. I personally think it is a bad tradeoff and will make developers hate their day.

Why would it be slow? Docker is just as fast as running native on Linux.

Unless you're talking about running it on Windows or Mac. In which case, don't do that. You don't want your developers developing on a different OS than your CI & deployment environment anyway.

> You don't want your developers developing on a different OS than your CI & deployment environment anyway

Eh, this kinda depends on what tech you're using. We use node.js at work which has very good cross-platform support. And we've had literally zero issues developing on mac and deploying to Linux in the 3 years I've been here.

Your (potential) problems will come at the installing node/packages (that might use C extensions or whatever) level, not developing on top of it.

Or just tooling issues like BSD vs. GNU versions, Homebrew installing things differently (e.g. using the more logical 'openapi-generator' vs. 'openapi-generator-cli' used by upstream and respected by at least apt, apk, and pacman).

Do you use Python by any chance? Node.js native modules typically bundle the native code as part of the module, and don’t depend on externally installed tools. I’ve never used OpenAPI generator, but that looks true of this npm package too https://www.npmjs.com/package/@openapitools/openapi-generato... (read me mentions the tool is downloaded and installed on first run).

I’ve had issues where modules didn’t run when upgrading OS versions (either macOS or Linux - but usually macOS), but we run prod on LTS versions of Ubuntu so that doesn’t happen very frequently, gets caught by our staging server, and is usually a case of simply upgrading libraries to newer versions.

I use both, and have had such issues with both.

I suppose using busybox as your base image will most readily highlight any missing library issues if you fancy it.

(If I remember in the morning I think I can quite quickly find whatever it was that was last a problem - it was definitely an npm not a pip install when last I encountered it.)

Anecdotally as well, we've had two that I can think of in the last two years (both were specific to Windows).

MacOS has a case insensitive filesystem (default). If you mix up the case with your files/folders and imports then everything will work just fine on your machine. We had a coworker that did this once. Stupidly, it even passed staging and somehow was merged upstream.

Indeed. This is the one incompatibility I have seen in the wild. On my previous machine I actually created a secondary case-sensitive partition to store code on. But unfortunately our iOS app doesn’t work on a case-sensitive file system so this meant having two different project folders. On my new machine I’ve not bothered and I’m just careful about casing.

WSL2 solved docker for us on windows (except broken opencl), so afaict just OS X now unless you go into major contortions

We do both containerized and native for developing one service, but generally docker compose for the composition (vs say k8s or all native). And important that CI is in docker: that means we can do accurate local docker testing of same env in case of CI fails.

While that's true for processes running in containers, building containers is generally noticeably slower than Makefiles as it involves several extra steps besides executing instructions (e.g. loading build context, gziping files, calculating hashes etc...)

You'd typically not rebuild CI images that often.

If you're following the article's suggestion of using docker to replace makefiles and/or run build processes in docker and just spit out the final product using --output, the you will build the image on every single compile. And I'd try to give docker credit for caching all the layers it can, but make caches results too so it's not necessarily even winning there.

You’re building Continuous Integration images all the time. The more often the better. Hence the aspirational “continuous” in the name.

> building containers is generally noticeably slower than Makefiles

I totally agree, if the year was 1992.

Compute is cheap. Storage is cheap. Let's not try and keep living in the past.

No, it’s noticeably slower than Makefiles on present-year hardware. This degrades the developer experience regardless of how cheap compute and storage are. (This is mostly Docker’s fault and not inherent to containers, though.)

> This degrades the developer experience regardless of how cheap compute and storage are.

I would argue that if you only have one good tool to do something, you have an opportunity to make much better tools for that something.

It's not docker's job to be better than Makefile with every use-case. You do realize what you're asking for here right? It's impossible. No tool is perfect for every use-case.

> It's not docker's job to be better than Makefile with every use-case. You do realize what you're asking for here right? It's impossible. No tool is perfect for every use-case.

The root of this thread is

> We actually use docker multistage build as mentioned here as a replacement for make file.

So yes, it absolutely is reasonable to ask that docker, when used in the exact same way as make, should not give us a noticeable performance regression compared to running make.

Did you even read the article?

345 words is barely an article. What about my comments are off topic exactly...?

Part of CI is fast feedback. Slow builds started soon still suffer from not providing the full experience.

It’s not a ratio that’s a problem, although that doesn’t help. It’s that there are psychological cutoffs where adding a minute to a build drastically changes how people interact with it. 1 vs 3 vs 7 vs >7 minutes make substantial differences.

So taking a build from 8 to 6 minutes is valuable out of all proportion to the actual speed up, and slowing from 6 to 8 is a substantial loss.

BuildKit(Docker build engine) maintainer here. It’s definitely our goal to limit the friction, so you get the benefits of containers but with similar performance as running things on the host. For the points you listed: BuildKit sends context incrementally between builds that transfers only deltas and not the whole context. Gzip is only needed when you want to push the image to a registry that doesn’t happen as often as your dev builds. For the hashes, we do need to calculate some for build cache (that of course makes your builds faster than they would be without it). For some bigger objects like layers we can defer checksum computation to push phase similarly to compression.

Another thing we have done is added cache mounts that let you keep application-specific cache between builds. This is usually, where the speed benefit from running things on the host used to come if your tools wrote some cache under $HOME etc.

There is still a constant container sandbox initialization time ~100ms, with some room to optimize that as well. If you have good examples of builds that can’t be easily optimized in containers would love to get your feedback/examples on the issue tracker.

> You don't want your developers developing on a different OS than your CI & deployment environment anyway.

Who's running OS X servers in production?

What are the devs running OS X and IntelliJ tools deploying on?

We have a pair of macOS servers in prod. But it's a weird edge case (we need to build iOS apps as part of the work queue, so there's no choice).

Mostly our devs run whichever of macOS or Windows is their personal preference (and in a few cases they choose Linux), and deploy to containers (or occasionally bare) Ubuntu or AWS Linux.

We occasionally get new devs bumping into the well known problems (case sensitivity, file path differences, some weird depenacncy that ends up platform-specific), but that's the kinda thing that is very easily debugged by someone who's made those mistakes before and even the greenest of new devs rarely make that same mistake twice (or at least they debug it themselves and don't admit to it).

IO (both network and disk) are afaik taking significantly roundabout way with docker vs bare linux

You can argue or disagree all day long about what OS is best for development, but it's not reasonable to think that developers on MacOS would only write code for MacOS servers.

You can very successfully write code that runs on Linux on either MacOS or Windows, depending on what you're developing.

Probably not a big deal, but just seems pointlessly narrow to think all developers would or should prefer actually developing on Linux exclusively.

I got frustrated enough with Docker on macOS that I now have an Intel NUC to use as my "local development". I still use my Mac for development (and all the tools I'm familiar with and do not want to learn replacements for) while also having "fast" Docker on my desk (and even when I'm not at my desk, Docker-on-Linux over VPN annoys me less that Docker on macOS).

Feels kinda wasteful sometimes to have all this compute horsepower in my MacBook, and then offload a bunch of the workload to a few hundred bucks worth of external hardware, but I reckon it'd saved my boss the entire cost of the NUC in the first week or two.

Not the right answer for everyone, but it works for me.

It's really not that bad of an idea for anyone, I can only speak for VSCode but what you're talking about isn't a huge gap from WSL2.

Except rather than using WSL2 for your local remote (confused yet? I am) sessions.

Hardware _is_ cheap, if MacOS can't support a decent enough environment for Docker, provide your Engineers with some cheap hardware like a NUC or even a remote AWS Linux EC2 instance or similar and get a native *nix environment that way.

You can but it’s easy to get your environment messed up with python.

Personally I’m a fan of buying a fancier Mac and running a VM that resembles production somewhat. I was using VMWare Fusion and now UTM. Works great for me by minimizing Homebrew drama.

> Docker is just as fast as running native on Linux

Once the docker image is pulled into cache, a runc process is started, that process is placed into a specific cgroup and given various isolated namespaces, any volumes are mounted into the file system namespace, any special network mappings are created, and otherwise, yes running within Docker is as fast as running natively.

> You don't want your developers developing on a different OS than your CI & deployment environment anyway.

It depends what you're working on, but for a classic server backend in a high-level language, I don't find it a problem. It's very rare to hit an OS-specific problem, and it shows up in CI so the dev can correct it before code review.

For me, filesystem I/O always ends up being significantly slower from inside docker containers.

Is there something obvious I may be doing wrong or some resources I should read about this?

It’s been a major issue for me. It seems to happen regardless of whether I’m using Docker Volumes or not.

I switched from docker desktop to Colima, my life has been much better since then.

> hard to debug

When I started using Docker four years ago that was my biggest issue. When builds fail, I can't just pick up where it left to expect the environment and try stuff out (unless I purposely build an image with a ton of layers).

Soooo much time was wasted figuring out what was going on.

Not to be reductive, but why not `docker run` the sha256 of the last successful stage of the build and run the next command manually?

That's what I generally do, but it's tedious as CI images are written to have as few layers as possible. So debugging often also involves changing Dockerfiles.

We had the some learning and decided to go down the path "Use docker for running, not building". That's why we decided to build https://bob.build on top of the nix package-manager. This allows reproducible builds on your system without touching docker.

I've been playing around with changing the default shell for some Makefiles to a shell inside a docker container (with dependencies baked in). It's a bit hacky, but kind of interesting!

[0] https://www.gnu.org/software/make/manual/html_node/Choosing-...

Can podman run within unprivileged docker nowadays ? (My main issue with those docker based builds is that they are annoying to run in unprivileged docker, in a CI for example)

You can actually [1]. See the last example.

[1] https://www.redhat.com/sysadmin/podman-inside-kubernetes

You should look into Earthly! They take this idea and turn it up to 11.

https://earthly.dev/

I needed to iterate on bare metal images. Docker buildkit turned out to be an awesome environment for composing minimal/fast vm/bare-metal images.

https://github.com/tarasglek/metallize/

Oh excellent, just the tool I've been looking for!

while it's probably much better than starting from scratch, might be a few rough edges.

I think Earthly is a better replacement for make; but containerized builds are definitely slower (often to a disappointing degree). Luckily Earthly has a ton of knobs you can turn re: caching to make things suck less.

https://earthly.dev

Earthly is my *favorite* tool. It replaces the glue of Makefiles fantastically.

Don’t get me wrong, make is great, but I think it only really shines when you’re using a language like C/C++ where you can use it’s awesome dependency graphs. For projects in Go or JavaScript it quickly becomes a task runner with a very inconvenient syntax.

Earthly gives you a make-esque syntax while doing all of the work in Docker which gives you repeatability and parallelization for free.

It might be a silly question since I know little about this tool but I didn't see a mention of docker-compose in the FAQ: do you have personal experience with use-cases where multiple volumes might need to be mounted, ports forwarded, and multiple containers talking to each others or depending on each others?

Earthly doesn't run containers, it builds them. You can use the resulting images with whatever runtime/orchestrator you want.

99% of projects I setup have a docker-compose.yml at the project root and it defines all of the external dependencies. docker-compose up -d and F5 in vscode… up and running.

Wish I would see more mentions of `podman` where you can just run a container without a daemon and all permissions issues (root by default in Docker).

Podman seems like a simpler container management tool for things that are merely wrapping/emulating traditional command line tools.

> As a cross-platform compatibility layer

We use docker in this way for work, and it’s merely okay. We use docker to bundle the environment (which changes rarely) separate from the code and build system (which changes often). For this, docker has been great for 95% of all our internal users: way more than a makefile helped, but the remaining 5% were a huge pain. The most recent issue was the in-docker user having a different id from the local user, which isn’t a problem on macOS but is on Ubuntu.

At the point where docker made sense as a compatibility layer, it was trivial to convert the whole system into a service running on Kubernetes.

> The most recent issue was the in-docker user having a different id from the local user

Probably the mount id mapping added in the linux kernel can be a way to resolve problem when user in docker and host have different id.

Although I don't know whether the system utils / docker command have integrated it or not yet.

https://lwn.net/Articles/837566/

I don't know about replacing Make with Docker, but I use the two together to good effect. One of my favorite hacks is adding a 'docker-%' rule in my Makefile to run make commands in a Docker image[1]. It's a bit mind-bending, and there's a few gotchas, but it works surprisingly well for simple rules.

[1]: https://github.com/kevin-hanselman/dud/blob/e98de8fcdf7ad564...

I toyed with a similar setup recently and I had the problem that I, too, would have rules depend on a "docker build"-step (like `docker-image` in your example). Usually `make` would stop building the dependency for non-PHONY targets if it finds the correct file but in this case it obviously cannot find anything.

I tried `touch`ing hidden files for each step and then add those as a dependency but that is not very elegant. Do you have this problem at all?

Make replacement? Why? Make is fast. Docker is not, unless you've very carefully set up the build stage pipeline--which generally has many more limitations than Make.

Because your Makefile likely depends on a very particular environment setup with specific versions of compilers, script hosts, tools, etc. You can meticulously maintain a giant readme that tells devs how to set each tool up (and inevitably takes everyone days and weeks of time to troubleshoot), or you can make a container that has everything setup and ready to go.

Make is great and you'd likely use it inside the container. But everything around make is typically an enormous pain to manage across multiple machines, multiple architectures, etc. and that's where docker containers help greatly.

Isn't it funny to see how folks have a hard time putting down the tools they're used to for the latest and greatest?

As if something being fast is all that matters.

Make also has more features for the intended purpose than Docker does—as I noted in my original comment.

Docker is still the right choice if you want environmental parity without needing to separately manage it. It's less specific and that's good.

Apart from caching, BuildKit also provides automatic parallelization. Because of these two features, we use BuildKit to make our research results, experiments with program analysis tools on software benchmarks, easier to share and faster to execute.

As a more expressive frontend of BuildKit, we created a simple, non-Turing-complete logic programming language, which can also be thought of as docker-based Makefile.

https://modus-continens.com/

I think this is basically the premise for Earthly and Dagger, right?

> That's part of the idea behind the co-founder of Docker's second act, Dagger.

Guess so!

Has anyone used both? How do they compare? I really like the idea of Dagger, but I found the configuration files impenetrable and couldn't even get started with a single build. I guess some of my issue was that I had to learn Dagger and Cue at the same time, and didn't know what needed what.

I haven’t used dagger, but I have used Earthly for a couple of months. It needs some polishing, but I feel like it solves 95% of my problems with Makefiles + Docker for containerized builds.

That sounds promising, thanks!

Same experience, I also looked at Dagger for it’s ecosystem of libraries, but I just didn’t understand it

And Toast: https://github.com/stepchowfun/toast

I don't know about BuildKit and `docker buildx`, but I've been using an approach similar to what is outlined in paragraph two to generate static executables for one of my projects.

I use a multistage build to generate a podman/docker image that contains all my build artifacts and then just copy them out of the container onto my host system. What advantages would there be in using BuildKit for this sort of thing?

Essentially the same thing in one step. Can avoid some of the permission issue gotchas with `docker cp`.

I can see the appeal as a replacement for make; in fact, it might be something I do without ever thinking of using make — just because I use Docker more often than make.

I use Docker for sort of a virtual network interface to allow me to connect to two VPNs (a VPN inside a VPN environment). Any good hardware/software finds all sorts of niches to fill in time.

Docker just announced Docker Extensions for Docker Desktop -- already configured long-running agents that are "managed" with a nice GUI.

I use the Tailscale extension to let my development pods talk to my tailnet. It's really handy (would be even better if it worked with the local Kubernetes cluster in Desktop). Of course you can do this yourself fairly easily, but it's nice to have it "managed" for a non-essential part of the workflow.

I would expect to see a code example, but overall makes sense.

Sorry, I should I have linked some in the article.

- Alternative docker frontends - https://matt-rickard.com/building-a-new-dockerfile-frontend/ (https://github.com/r2d4/mockerfile)

- Docker merge - https://matt-rickard.com/docker-merge/ (https://github.com/r2d4/docker-merge)

Cross-compiling binaries for many platforms and architectures using Docker and QEMU[1] was a neat way I found to support more targets without having to have the hardware.

[1] https://www.qemu.org/

Docker doesn't really help with ARM/Intel differences. You need something like JVM for that.

I like, how someone solves the OS issue and suddenly people start using different CPU archs.

It actually does nowadays. The --platform flag integrates with QEMU. It's emulation so it's slower, but better than doing it manually.

[0] https://docs.docker.com/build/buildx/#build-multi-platform-i...

Either QEMU itself or the QEMU integration seems a bit buggy to me coz it goes wrong a lot.

I use QEMU on a regular basis and have hardly ever have issues so I'd guess it's the integration piece.

It does help, docker's buildkit backend (enabled when you use the buildx command instead of build) supports building for other architectures automatically. You can for example on an ARM device build a perfectly fine x86 or x64 image and push it to a registry, or vice-versa. It works by behind the scenes using QEMU to emulate the other architecture and binfmt_misc to run its executables. This all happens transparently and wraps up what used to be a bit of manual work and tweaking.

It also supports running other architecture images using the same QEMU magic. It's really quite polished and works well.

Docker supports multi-platform builds, and many popular images have built in support for both architecture via their manifest. In my experience this only breaks in your own images where you download and install some package with a URL with a hardcoded architecture.

Docker does provide environment variables you can use to automatically download the right file depending on the target architecture, but the syntax is very cumbersome.

That said, yes using something on the JVM is much more convenient, but that’s not possible if you’re not using a JVM language.

I have been using Dockcross for years for cross-compiling

These do look a little like "an alternative use for a screwdriver is to use the handle to bang in nails". Yes, but it looks a little hacky and I suspect there might be another tool designed for that niche. I'm not convinced that the screwdriver development team will be prioritising its nail-banging feature long-term.

any examples of the "registry as config store" idea? i've got a use-case where that could be pretty handy

I can't find the code anymore but I originally saw this in docker/compose.

I forget what they were using it for, but it boiled down to packing up a json file or two in a tarball and pushing it to the local daemon with a special suffix. Then it could pull that and unpack it on the next run.

Note that with "Docker as package" style development, you lose the ability to specify dependencies. Consider building real packages, then installing them in shim Docker images. This gives you the best of both worlds: rich dependency specification, and "just run the thing" facility with Docker.

There's lots of examples of folks packaging non-container-image things and stuffing them in a registry:

- cosign signatures, SBOMs, attestations: https://docs.sigstore.dev/cosign/overview/

- Istio wasm plugins: https://istio.io/latest/docs/reference/config/proxy_extensio...

- OPA bundles: https://www.openpolicyagent.org/docs/v0.13.5/bundles/

- Tekton bundles: https://github.com/tektoncd/resolution/tree/main/bundleresol...

There's even a demo (plug: in a talk I co-presented) of running a RISC-V emulator where its memory is stored in an OCI registry: https://www.youtube.com/watch?v=Xt_G-pUArTM

These all tend to include a CLI to collect/generate/validate resources and push/move/pull them, but the underlying implementation is roughly the same -- package content in a tarball, generate some JSON pointing to it, push that JSON to the registry (with auth).

The real benefit to this is that basically everyone has access to a registry these days -- in their cloud provider, on-prem, whatever -- with exactly the same APIs and mostly sane auth and client tooling.

If you're interested in exploring it for your use case let me know, I'd be happy to give you some pointers.

Helm can store kubernetes charts in an OCI/docker registry too: https://helm.sh/blog/storing-charts-in-oci/

thank. helm links to https://oras.land/, which seems like a pretty cool generalization of all this.

I gave a very compressed lightning talk last year about this: https://youtu.be/ExyWAhS2zBA

Something about a hammer and nails.

When the options are a hammer or a book of secret spells...

Reply on news.ycombinator.com