Anti-patterns when building container images
jpetazzo.github.io
jpetazzo.github.io
They had a brillant sysadmin, that I bothered to explain them to me. Sometimes, he'll come to my desk, looking at my screen, typing the solution while explaining the problem to me with simple words.
Standing up, typing with one hand, and working on what was supposed to be my specialty, he was doing a better job than I was, despite me interrupting his workflow.
I left the job, and levelled up. One day, I read an article about docker, and the name of the dev caught my eyes: it was the friendly former sysadmin colleague, Jérôme Petazzoni, who went on becomming docker's biggest evangelist.
Thank you Jérôme, for making sure I started my carrer by not being the smartest guy in the room. I kept it as a habit.
Or I'm just dumb :)
That's indeed invaluable. That should be the first advice to anyone starting out: if you find yourself being the smartest in a room, go find a new room. Right now.
If you are ever managing someone who is "your best employee" and they want to do something you think is useless, please ask them why and what the expected results will be. Also when you get that data, listen a little? If priorities clash do a balancing act: "If you can do X in Y time, do it. But if day D comes along, switch over to my priority".
For instance, they required that all "software engineers" were from an accredited engineering program--at the time, none of the universities in the area had an accredited software engineering program, so they would hire electrical engineering majors who had no desire to write software but were looking for a way to get their foot in the door. Most of the EE majors had a single semester of C and that was their entire software qualification; however, I double majored in EE and CS and my passion was software. That said, when I graduated I didn't understand any of this--I just figured there was this successful company and there was so much I didn't know about software, so I thought I would learn loads there--and I did learn loads, but it was mostly about what not to do.
One of my first projects was to take over an embedded C++/Qt project that had been bid out to some contractors in the Ukraine. The software was super buggy and the code was immensely convoluted. Test coverage was part of the contract, so it had tests, but even still, no one could change anything without breaking everything. It took me weeks of plotting the code paths to figure out that there were many paths through the code that were only exercised by tests--literally they were just trying to boost their code coverage metrics by adding code that was unreachable at production. I guess it didn't occur to me that professional software engineers would be so bad at writing code, and that a large and well-known software company would hire them.
Thankfully, after eventually gaining my entirely-nontechnical manager's trust, we were able to bring in a consultancy who helped me demonstrate that it would be no more than half the cost to rewrite the code rather than try to fix all of the existing bugs even if we never found other bugs or introduced a single bug of our own. We did the rewrite and it was successful, but I was beginning to realize that in some overly simplistic sense, I was the "smartest" one in the room as it pertained to software development anyway.
Ten years later, when I talk to my peers who stayed, many of them have been writing embedded C programs and various MatLab models and they still don't understand pointers or dynamic memory or git or anything outside of their very local niche. Which would be fine if they were happy, but now many want to move to another company but they lack the skills and confidence to get anything more than an entry level job at many other software companies--so much of their compensation is derived from their knowledge of their employer's business and processes rather than their technical acumen, so they would likely have to take a significant pay cut to move to another employer.
Anyway, this is already a very long post despite oversimplifying many things, but I hope this illustrates what it looks like to be the smartest person in the room and why it's not good.
Information like this is very dependent on the context of when it was written, and something that is "best practice" or "an anti-pattern" will not always be that way. Adding a publication date and/or a "last updated" date to the top of the article is very helpful for readers to align the information in their minds.
EDIT: I realized just after posting this comment that the publication date is in the url. Time for that second cup of coffee :)
<!--<p class="meta">30 Nov 2021</p>-->I'm always a bit curious why more people don't talk about using nsenter in the rare situation they need to inspect stuff in a running container?
For a production microservice container, why have a bunch of bloat in the container that increases the surface area to manage while also only actually being used in edge cases?
But I don't aim for minimal images. Using alpine and installing what ever you like without worrying about all the bullshit to make caches stay empty still results in images only a few megs, don't need to do much better than that.
Though I tend to agree, if you're talking about a compiled language. Static binary and there is little need for anything other than `from scratch`.
As for the status quo, I spend a lot of time extracting dumps from things in containers and debugging weird issues which is really really difficult without the ability to install any tools on the container at will. When done, we blast the container and let kubernetes reschedule a new one, disposing of all tools and state.
On bloat, mine are sitting on top of a debian container layer, mostly because it’s a hell of a lot easier dealing with some dynamic libraries than fecking around with static linking stuff and copying dependencies into a minimalist container.
Edit: There is overlap between these two things which nsenter may not be able to solve looking at what I am reading.
In theory. In practice this is only rarely the case.
† https://kubernetes.io/docs/tasks/debug-application-cluster/d...
Then when you pull the Docker image onto a machine it’s likely the base image is already in cache so again it’s only the diff that is transferred.
It’s sometimes said this Alpine stuff is a premature optimization, but in most cases it’s not even an optimization.
Not always. If you have thousands of containers running, the network cost* for changing the base image is massive because each instance needs possibly all new layers, not just the last layer or 2. Having a smaller base image is very practical at this point.
* And for network cost, there is both the literal cost with egress from dockerhub, or another registry, but also the load on your network infra.
Also, Alpine images are used for being, well, small in general - there's not much to exploit on such image, nor is there going to be a lot of unplanned-for data, and especially it will speed up deployment when you upload a new image.
It takes special, dedicated CI/CD work to have common base images to exploit sharing of layers properly.
In my job, I rarely have an exact match on the base layer because I'll use images from Docker Hub. Although many of those images use the same small set of distros (Alpine, Debian, or Ubuntu), any given image is likely from a different base version of that distro, thus there is no common base image.
There may be differences in the base image even if you use the same tag since the same image may change over time. Thus if you pull debian:bullseye today, and then pull again tomorrow, you may have two different images.
Generally I don't bother trying to get a common base image and I'm happy enough just attempt to lower the overall size of my images.
I do use Debian Slim over Alpine without thinking but image size does matter in a lot of cases once you leave your dev box or start to use Kubernetes.
For example if you're using AWS Fargate images aren't cached[0]. That means every time you deploy your app a new pod is going to get created and it has to wait for your image to download.
Fargate already has a 30-40ish second spin up time inherit to the platform. Having to wait X seconds or even minutes more to download your app's image delays things quite a lot, especially if you have something like a pre-sync hook running to do something like running a database migration. Now every deploy pays this penalty twice.
Consider that busybox is just under 1MB when dynamically linked and installed. That's roughly 50% larger than a distroless container.
Further, distroless doesn't just target size for size's sake - it's also to ensure that an attacker in the container doesn't have tooling that they like to use.
Instead if your containers are mounting a volume, say `/var/run/debug-tools/`, you could copy the tools into that mount before entering your containers. This probably has a good dev UX in that you can enter containers and have your tools, it amortizes the cost across your containers on a given host, and it isn't container specific ie: you perform one copy and now all containers on that host have it.
That is, if we're talking Kubernetes of course.
Otherwise you have to prepare in advance and add a small sidecar to all pods that need it
I would add that using Docker in the first place is an anti-pattern. If you're using software you wrote yourself that can be made self-contained, e.g. if it's written in Go or Rust, then you probably don't even need Docker.
People always say "but how can you isolate the network?" and use Kurbernetes and .. and sure Docker is the easiest way to do that. But in my experience people rarely use Docker for that feature anyway. 99% of Docker use is because most Linux software is written in such a way that it is an absolute pain in the arse to distribute so shipping a whole OS is the bodgy solution that everyone accepts.
I tried this recently with `kubectl cp` for my scratch image and it turns out that command will fail if the target container lacks a `tar` program in its PATH. I guess under the covers it's just doing a `kubectl exec <pod> -- /bin/tar ...` on the file rather than asking the kubelet to untar the file into the target container's filesystem.
> Big images
> Requiring rebuilds for every single change
> Forcing things to run in containers
> Using overly complex tools
> Building with Dockerfiles
They make it clear what steps occurred to arrive at the resulting image. For any image that is built from an open-source repository, I can find the Dockerfile, look at it and say "Oh, that's how they did it!"
In the article it says that Dockerfiles might not be the best way to build your image. I think some of that may have come from before the days of multi-stage builds. With multi-stage builds you can use a lot of tools (like the Go compiler) at build time without including those tools in your final build.
Can we do better than Dockerfiles as they are today? I'm sure we can, but can I appreciate what I have while knowing it could be better.
I don't like Docker as a company, but I can't imagine living my tech life without containers + Kubernetes, and Dockerfiles are really easy to write and read.
Unfortunately avoiding the anti-patterns seems to be much harder than adopting them.