Comment by hinkley | Hacker News Reader

Comment by hinkley | Hacker News Reader

Parent Full thread

This is something that even the Docker core team is not entirely clear on (either they understand it and can't explain it, or they don't understand it).

The RUN command is Turing complete, therefore cannot be idempotent. Nearly every usable docker image and base image include the RUN command, often explicitly to do things that are not repeatable, like "fetch the latest packages from the repository".

This is all before you get to application code which may be doing the same non-repeatable actions with gems or crates or maven or node modules, or calling a service and storing the result in your build. Getting repeatable docker images usually is a matter of first getting your build system to be repeatable, and then expanding it to include the docker images. And often times standing up a private repository so you have something like a reliable snapshot of the public repository, one you can go back to if someone plays games with the published versions of things.

pedantic nit-pick that doesn't detract from your main point: Turing completeness doesn't imply that it can never be idempotent. In fact we'd expect that a particular Turing machine given a particular tape should reliably produce the same output, and similarly any Turing-complete program given the exact same input should produce the exact same output. Turing machines are single-threaded, non-networked machines with no concept of time.

The main reason the RUN statement isn't idempotent, in my opinion, is that it can reach out to the network and e.g. fetch the latest packages from a repository, like you mention. (Other things like the clock, race conditions in multi-threaded programs, etc. can also cause RUN statements to do different things on multiple runs, but I'd argue those are relatively uncommon and unimportant.)

Yup, the problem is not that RUN is Turing Complete the problem is that it’s non-hermetic.

If it's Turing Complete you can't prove that it's hermetic. You can't even prove if it'll ever finish running (halting problem).

Maybe someday someone will invent a sort of EBPF for containers, but usually the first batch of RUN commands and the last are calling package managers, and in between you're doing things like creating users and setting permissions using common unix shell commands, which have the same problems.

With the possible exception of the unix package managers, we have no hope of those ever being rewritten inside of a proof system, because the package manager is often a form of dogfooding for the language community. The Node package manager is going to be written in Node and use common networking and archive libraries. Same for Ruby, Rust, you name it.

> If it's Turing Complete you can't prove that it's hermetic

Turing Machine model doesn’t apply to i/o so it is a meaningless statement.

Bazel and Nix achieve this by sandboxing i/o no turing incompleteness needed (tho bazel is still purposely non-tc but for totally different reason than hermeticity)

>Getting repeatable docker images

But there's no real need for repeatable build docker images. You copy the image and run it where ever you need to. The entire point of Docker is to not have to repeat the build.

If you could explain that to the Docker team you would be performing a great humanitarian service.

They for many years close issues and reject PRs based on the assertion that Dockerfile output needs to be repeatable (specifically, that the requested feature or PR makes it not repeatable). It’s not, it can’t be without a time machine, and if it was I believe you’d find that ace icing traction would have been more difficult, even impossible.

Do you have an example?

Until there is a critical vulnerability in one of the components present on that image (system packages or application packages).

This is why I haven't adopted the practice of "Build your artifact along with the docker image that packages it".

Instead, build your artifact and publish it to an artifact repository, just like we used to.

_then_ wrap that artifact in a Docker image.

Vulnerability found in the docker image? No problem. Build a new image with the same artifact.

I'm curious why you're being downvoted for this. Whoever disagrees, please share some context.

What you said about "wrapping" I interpret as: based on an image for instance with OS level dependencies you create another image with application level artifacts, e.g. a python application. When your app changes, you don't build the base image again, you only build the app image. This makes sense to me.

At that point you're not repeating a build. You're building a new image.

From that point on, the build may become "fully unrepeatable" because previous steps may have reused cached stuff that can no longer be reproduced.

You could break the build simply by cleaning some things from the cache, without changing the Dockerfile, even thought that Dockerfile "succeeded to build"

what happens when you need to create a new one with a small incremental change?

Reply on news.ycombinator.com