Reducing Docker Image Size | Hacker News Reader

Reducing Docker Image Size

ardanlabs.com

176 pointsaberoham35 comments

Seems to be a pretty decent overview; covers the usual suspects (multi-stage builds, FROM scratch, non-scratch minmal images, ldd to check libraries), with some nice bits that I'd not seen before (busybox:glibc). I would be curious to see how these base images stack up against Google's "distroless" base images (https://github.com/GoogleContainerTools/distroless). I also appreciate that they call out Alpine's compatibility issues (on account of musl) but still leave it as a thing that can be good if you use it right. (Personally I'm quite fond of Alpine, but I don't bother when using binaries that expect glibc.)

One important thing missing from BusyBox:glibc is the SSL certificates that's almost always needed if you are writing things like crawler etc.

As for Alpine, at least for packaging Python code, avoid as much as possible. It cannot reuse manylinux packages on PyPI and have to recompile C modules.

With Alpine, the size advantages of musl sometimes come with hidden overheads as well.

Musl's malloc() seems to be a fair bit slower than glibc's when you start to throw in more threads at it (fragments faster?).

And then if you try to replace that with jemalloc, you'll quickly find out that jemalloc does not actually get used.

At some level, the difference between a 1Mb docker image and a 9Mb one isn't that significant (& I asked for some things like strace, ping and nslookup to be in it, because it gets impossible to debug the container when it obviously caches a DNS record or at least, looks like it is doing that).

and as a result: plenty of breaking builds https://github.com/pyca/cryptography/issues/4264

One thing they don't talk about is using non-docker tools to build images.

e.g. nix: http://lethalman.blogspot.com/2016/04/cheap-docker-images-wi... or jib: https://github.com/GoogleContainerTools/jib

These may not fit every use case, but they can minimize the size of the image by making really precisely defined containers.

It’s worth looking at an overview of how Jib works because the general strategy is very good IMO.

https://phauer.com/2019/no-fat-jar-in-docker-image/

been using docker-slim[1] for a couple years and am really happy with it.

Also I build everything from source and try to avoid glibc wherever I can, use hardened malloc and -D_FORTIFY_SOURCE=2 everything. also use static linking wherever I can and try to --disable-foo with configure if I can get away with it. Using a minimalistic approach that restricts everything (down to a syscall whitelist) sounds like it's high maintenence but when done in combination with a catalog of what runs in these containers and some plumbing to poll upstream for repo updates allows me to ignore a lot of what affects most people wrt security. As a bonus I learn about it quickly if upstream introduces a major change and I'm forced to study the source and get to decide if they were "out of their mind again" with these changes.

Keeps me abreast of how my platform behaves and when it doesn't as it should (it's kind of a forward investment for efficient root cause analysis that beats "have you tried turning it off and on again")

Docker imo doesn't deserve the hype. Most of the flaws I deal with in my setup are due to docker instability (often in docker swarm) ... it's the next thing I actually want to replace because it's just overhyped "static linking for Millennials".

[1] https://github.com/docker-slim/docker-slim

Docker Swarm has been pretty much deprecated.

When Mirantis bought it, they said they'll support for 2 years but Kubernetes is their focus.

thanks, a good reason to ditch it then.

While such articles are usually helpful, I'd caution that making individual image sizes as small as possible shouldn't really be your goal.

As a simple example - if you have 100 unique images in your system, having an image size of 1 GB each where 99% of it is derived from a common layer is going to be a lot smaller overall than "optimizing" the size down to 100 MB each but taking away the base layers.

In my experience you 100 unique images might still have different base images. So you end up with too many layers. Especially with interpreted languages those python, ruby or node runtimes add up quickly

Also depending on usecase (e.g running a CI system) you might end up with high storage costs (caching) or high bandwidth costs (no caching)

Depending where your datacenter is, coworkers with slow connections might suffer as well. Hey, let’s just download 12GB of my everylanguage-and-Tex:latest every workday.

But I agree, docker golfing to have minimal images is not always needed, but don’t let your images grow like a wildfire. They are part of your architecture.

Using a trendy small base image doesn't make the intermediate layers any smaller.

What you're describing is what cloud native buildpacks are intended to solve: sensible, efficient, auto-updateable OCI images. I worked on them for a while and it just hardened my heart against Dockerfiles.

It's starting to feel like this is a lost battle though. People have decided that Dockerfiles are "the kubernetes way", even though it exposes developers to details they just shouldn't need to care about.

Add anything on top, and it is no longer "vanilla kubernetes". Never mind that means everyone roles their own Rube Goldberg build process, and somehow that is ok...

I don't feel like it's a lost battle at all. Folks are fanning out and learning that all of this is harder than it looks, with the exception of enterprise, where it's harder than it looks squared.

I suspect folks will become attracted to the simplicity and performance of buildpacks. They make whole classes of headache vanish.

I guess on this one I'm optimistic.

A standardization for base images, OS etc. is just something that has to be enforced at the organization level.

But if you do that you're giving up all public images. It makes sense in a large organization but there are plenty of other use cases for Docker.

If you are using public images, then you are already giving up on optimizing image sizes anyways.

But if you are making a public image you are doing your users a favor by minimizing the size.

You don't have to give up public images. Here's a simple example rule - production images should all be built on Ubuntu:18.04. (With a large enough maintenance team this can change to - you must use one of these 8 images with the standard runtime stacks preconfigured).

Teams still have the flexibility of using whatever they want during the build stage, and it takes away the ambiguity of them having to decide between different OS flavors, versions, slim/Alpine, building from scratch or whatever else. In 99% of cases there will be no functional difference between them anyways, and this rule reduces the overhead of having to track down and manage an infinite number of permutations from an operations side.

Almost every CI setup completely breaks that assumption since layers are not cached efficiently among job runners.

Why is image size important? Should we instead optimise for speed of build?

If storage is cheap. And CPU costs Co2 does it make sense to spend longer time and more energy to save disk space?

>Why is image size important?

Cold start times I'd imagine

It's not just the raw size of the image, but also about what the image includes; a smaller image often reduces the potential attack surface because vulnerable things just aren't there.

That's one of the major rationales behind the distroless images. Being space optimized is just a really nice side effect.

>> a smaller image often reduces the potential attack surface because vulnerable things just aren't there

By the way, the article proposes blind download of artifacts from someplace on the internet, on every build. Not only that can cripple your builds when the source is down (which happens all the time), it can (and that has happened) send you arbitrary infected crap instead of what you wanted.

There are plenty of resource limited use cases, where storage is not exactly cheap. Or updating image over network might be slow or expensive (think edge, over 3-4g)

How many servers are you copying the image to?

security (e.g. attack surface)

It's worth noting that golang builds can be smaller than that with `GOOS=linux go build -ldflags="-s -w" .` (assuming a build on macos for linux.) From there I usually run `upx --ultra-brute -9 program` before dropping it into a `scratch` docker container (plus whatever other deps it needs).

Does the UPX help much on top of Docker's layer compression? I can imagine it might (special purpose vs general purpose) but haven't tried it.

Just saw this reply, yes it does.

This is indeed a good overview.

It would have been a great overview had it started with briefing readers about why (or when) image size should bother us at all.

Are containers really the best abstraction for providing application isolation / portability? What about snaps/nix/guix instead?

They're probably not best at any of portability, isolation, or security (they probably lose to full VMs on all counts). But they're good enough and way more convenient / easier to use.

Strange advice to not use Alpine, but then recommends an empty image or busybox...

I'm pretty sure that's 100% a glibc/musl thing.

Reply on news.ycombinator.com