Just say no to :latest
platformers.dev
platformers.dev
Pinning versions creates work in the future. Using latest versions can result in ad-hoc breakages and security risk. So instead of saying "v1.22=good, latest=bad" and talking generally about the fact that lock files exist in most decent systems, I'd like an article that contains example strategies for paying back the tech debt of a thousand arbitrarily pinned versions across your code projects.
I like the idea of deleting the lock file on a regular cadence, accepting whatever new shit comes in, perhaps bumping a major version number, and then testing the crap out of everything and leaving this new artifact in integration/staging for a few extra days. If things go wrong between builds, then you have the lock files in git (and in your artifacts) to let you know what changed. If it turns out that you now have a real reason to pin a specific package (ie: it broke something), then you have to find some way to note that outside of the usual lock files, since they arbitrarily lock everything.
When evaluating new toolchains, I often ask myself "what is the versions plugin equivalent here". There's often commercial offerings that help this process out.
Dependency management is one of these topics that rarely gets attention until much later in the ecosystem's lifetime. So chances are you'll need to find or create tooling to fill gaps.
I mean, you can do the whole "damn the torpedos and see what breaks" approach, but, I'd prefer a way to review changes. And I'd prefer doing those updates _separate_ from other changes.
Why? If your lockfile specifies and enforces the hash of the package, you don't need to self-host it - the lockfile will take care of detecting supply chain compromises that swaps a package with another.
100% agree on audits being necessary.
If you really want to be sure, having the exact bits on a system you control seems like the right move.
Conversely, there's also a factor that says that after some time, the security community will stop looking at a specific version. Even if that version still has vulnerabilities.
Those two factors lead me to believe there's some sort of sweet spot. You probably don't want the cutting edge newest version, but you still want a version new enough that it's in widespread use.
This is a pick your poison circumstance.
IMHO a reasonable compromise is to watch for discovered CVEs and update your dependencies when appropriate, but keep them pinned if no known CVEs apply.
Vulnerability discovered in old version vs. vulnerability maliciously inserted in new version; software engineers afraid of upgrading because of breaks vs these breaks actually occurring because of an unexpected upgrade.
I personally prefer pinned versions because whenever I try to install some old github example or other code, I usually end up getting stuck with errors unless everything (compiler, dependencies, etc.) are pinned. Gradle projects almost always work no matter how old (Gradle pins the dependencies, JVM, and even the version of Gradle), whereas npm and CMake projects (which are built with new compilers and usually new dependencies) often fail.
Then the tech debt gets addressed automatically without any work. The work is all up front to write the tests that give you confidence about deployment to production, but that is work that needs to happens and pays off. You shouldn't need to have human testing in the loop.
If your requirements aren't that sharp, like maybe you can take a day or two to process the credit card payments, you could get away with monitoring your application and configuring alarms for if no transactions go through. That way you'd catch errors originating from your third party as well.
https://www.whitesourcesoftware.com/free-developer-tools/ren...
FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:15c22c170650b8db2f6250547a2dc5341978b0647c6b21ef67768e628de614f3 AS build
So a when upstream updates their tag I get a PR(which is automerged) that looks like so, this allows me to know when upstream has changed while still being able to target a broader version range, 6.0 in this case
- FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:15c22c170650b8db2f6250547a2dc5341978b0647c6b21ef67768e628de614f3 AS build
+ FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:70b890cd12f73f8ad80061d242081b61da666bda7ec2d729113855a8b9410e1e AS buildRobust systems need to model both so they can support reproducibility and automation needs.
This security scan would be absurd. The amount of churn of updating everything the day it is released would be insane, specially wrt. breaking changes.
We’ve found many broken builds thanks to this, and could then create an issue in our tracker to tackle that tech debt later.
If the build doesn’t break, we review the new version’s release notes, and sometimes even look at the code changes of the open source package. Overall it’s helped us keep more up to date, has made upgrading versions of Node less painful, and even helped us adopt some improvements sooner than we otherwise would have.
If I have project that has good unit test coverage, I prefer to use :latest, as this results in a gradual update over time. If something breaks due to a version discrepancy, it is a lot easier to convince management to fix this, as the breakage would only be noticed as part of a feature request, and often would only require a small amount of work.
You need vulnerability & risk management for any dependency, full stop.
It is always a trade off, however it is far more likely that a hacker will use a ten year old well exploited CVE, rather than a recent one
Docker images don’t ship every dependency in the average development project. They’re also not a security guarantee either.
> It is always a trade off, however it is far more likely that a hacker will use a ten year old well exploited CVE, rather than a recent one
In general that’s the case but in practice that’s still the wrong mindset. You pin to a minor version and still get patch updates. Or run CVE checks to prove you’re dependencies have no reported vulnerabilities. And that will also mitigate the risk of you again accidentally updating to to new vulnerabilities (though you’d need to pin against patch version too if you want to be certain there)
This is about succession planning, with the realisation there might not be any succession if business priorities change.
Ultimately there’s no perfect answer here but given a choice I’d rather opt for a known risk than an unknown one.
FROM python:3
If, however, one wants automatic updates WITH reproducible builds, then a CI/CD pipeline that automatically updates the FROM line in a Dockerfile on every upstream release is an only solution.
FROM python:3.10 should generally be fine though.
The PyTorch scenario is an exception, not the rule, and the users of PyTorch should know not to use unsupported versions.
> For anything nontrivial, you almost always want to specify a minor Python version.
Depends on how you define "trivial".
Besides, if the dependencies aren't pinned in version control, how can you answer a query like "On this date last year, what version were we running?"
To give an example, we got stuck on Python Pandas 0.24.2 for two years when the Pandas 1.0 upgrade happened. At the time we pinned the version as one job was failing. With COVID priorities naturally changed and this was forgotten. When normality returned we suddenly had to upgrade 10 jobs to Pandas 1.2.x, a very painful migration that took one Engineer 4 weeks of testing. In hindsight it would have been far better to upgrade this one job, as it would have only taken a few hours.
Again it is a trade off. Having unexpected dependency changes can be cumbersome when you are releasing a small change. However, getting senior management to agree of weeks of upgrade testing can be challenging as well. My natural preference is not to pin versions, unless I absolutely have to.
- having tests (or even manually test the system with scenarios, if for whatever reason you cannot automate) in place to catch things breaking, before shipping any changes
- having security scanning in place, be it for containers, your dependencies, or anything else; ideally for everything
- you should be able to spot the publicly known vulnerabilities and adequately evaluate their impact to decide what must be done
- when you update versions of your dependencies, you should then also be able to see whether anything breaks before shipping
Not only that, but some systems out there are not really easily testable, e.g. those that have a large amount of functionality within the DB so your tests might as well end up mocking either too much of the system or the wrong parts of the system (i'm yet to see anyone mocking the low level queries that go to the DB and back, e.g. setting and validating the individual parameters within query abstractions, as opposed to just the data that's returned) - in many cases it simply won't be viable to test everything.
Another thing is that in practice semver tends to lie to us and even minor updates or patches can sometimes have breaking changes within them, something that i wrote about in my blog article "Never update anything": https://blog.kronis.dev/articles/never-update-anything (albeit there i also touch upon the fact that software should largely be more stable and have fewer breaking updates in the first place, or even less new features in "stable" branches)
I guess my argument is that there's a lot of complexity to be tackled here and that people should invest more time and effort into handling updates, refactoring and even testing, than they do now: i've seen teams where people all agree that tests are important, yet nobody wants to write any because if they tried, they'd have to mock large parts of the system OR try to do integration tests and handle the fact that nobody has invested the time and effort into bringing up reproducible environments and services, e.g. a new automatically migrated DB instance for the tests OR the fact that they'll need to do tests in a shared DB instance and clean up afterwards. Environments like that are just a downwards spiral that's bound to produce brittle software.
So what's my practical advice?
Use pinned versions, preferably starting with the latest and most boring one that you can get away with (e.g. LTS). Have a process in place for figuring out when you need to update, set aside time for doing just that (even if manually), assign someone to do this and someone else to make sure it's been done. Have testing and serious validations be a part of this process, to make sure that there's no breakage that'd slip past - full regression testing, all your unit tests, integration tests, feature tests etc. How often should you do it? Depends on the importance of the system - i've seen it done quarterly, i've seen it done monthly, some with plenty of resources out there could probably do it weekly or more often. Additionally, be able to do this ASAP when critical vulnerabilities become apparent, e.g. log4shell.
Of course, it's the ultimate example of useful but exceedingly boring work that people don't really want to do, so i've no doubts about running into unmaintained software in the future.
I personally think running latest is the best thing to do. And if something fails, you downgrade it temporarily until the latest work again. It's pretty much opposite to what is recommended, and it's just the best solution in my opinion.
I'm only half joking... The right thing to do with dummy management is to get everything in writing to make it clear that if anything happens it'll be their fault.
(Jackson, take a bow for doing so with your patch releases. I wouldn't use it for your 'minor' releases though.)
1) JVM libs generally tend to follow semver to the Y level of X.Y.Z. reasonably well. (Spring can FOADIAF in this regard though)
2) Maven and Gradle etc. have very good dependency resolution algorithms
3) When the dependency algorithm struggles, you can provide guidance (e.g., Maven's dependencyManagement element, or the ability to exclude a dependency's dependency so that your pinned version of that dependency is what's used)
On that last point, you can use the Enforcer plugin to be very very draconian about transitive dependencies. I worked on a project in the past that would fail the build if you introduced a dependency that had dependencies, and didn't precisely specify the version of each new dependency at the project level.
I've done a fair bit of work also with Python, Go, and JS, and doing so made me realise that the dependency management situation in JVM land is the best I've ever encountered (so far). Which surprised me, I had always thought it somewhat over-engineered, but now I think it's sufficiently engineered and those other languages need to steal some ideas. (Especially Python, I love Poetry, especially compared to pipenv, but I would kill for the ability to override a dependency's dependency like I can in Maven and Gradle).
Best to upgrade and discover dependency issues during dev/staging, than try to fix many months worth of them when there's an active CVE on your live pinned version.
Too many times Ive seen companies throwing this buzzword around without understanding how much work is actually required to have real devops.
- https://github.com/hadolint/hadolint/wiki/DL3007: Using latest is prone to errors if the image will ever update. Pin the version explicitly to a release tag. - https://github.com/hadolint/hadolint/wiki/DL3013: Pin versions in pip. - https://github.com/hadolint/hadolint/wiki/DL3018: Pin versions in apk add.
Too slow - Exposed to known vulnerabilities.
Too fast - Susceptible to active supply chain attacks.
There's tradeoffs, and there's no solution here. It's a hard problem.
1) Just because there is a vulnerability in a package of library doesn't mean that you are susceptible to that vulnerability 2) If you don't update immediately when will you? Most companies can't afford people to understand all posted vulnerabilities to work out whether updating is better than not updating
The truth is there are easier ways to reduce risk than pretending that a pinned version somehow solves a lot of problems: Reduce your codebase/dependencies and apply defence-in-depth.
Reproducability is something for some companies but, again, I doubt there would be many times that I would care about what exactly was deployed 6 months ago. If something is broken now, it gets fixed now.
It doesn't solve the "immutability" problem and may give you a false sense that it does, which is a much bigger problem.
Here's the "good" example from the article:
# GOOD:
image: "nginx:1.21.6"
FROM debian:bullseye-slim
LABEL maintainer="NGINX Docker Maintainers <docker-maint@nginx.com>"
ENV NGINX_VERSION 1.21.6
ENV NJS_VERSION 0.7.2
ENV PKG_RELEASE 1~bullseye
...
...
...
If you are serious about using docker, you need a private registry and you need to make a tagging policy that is best for YOUR use-case.
For example
FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:70b890cd12f73f8ad80061d242081b61da666bda7ec2d729113855a8b9410e1e AS build
git push -> Docker build (bonus points for building only if needed [for example only if Dockerfile, Jenkinsfile or requirements.txt has changed], otherwise use latest from Artifactory) -> run all automated tests -> if pass, push the Docker image to Artifactory with reasonable name and tag.
When doing a release, push the image from Artifactory to production.
This way you don't have to have a process to update all the things in Dockerfile on reqular basis, but you still only push to production the actual binaries that are really tested and proven to be ok. And can re-use the actual binary when doing a hotfix of something similar.
It's the same with locking python pip packages to specific versions. Nobody ever looks at it again and you run code that is 5 years old in production. People only look at it when it breaks.
If you lose :latest, and the build breaks, you need to spend unexpected time fixing the build, for whatever reason that :latest broke it, in order to continue with the business objective that you were actually tasked to do. Essentially, you have InfoSec muscling in and demanding, by the most coercive way possible, that Thou Shalt Not Do Your Job Without Bowing Before Me. Who cares what the business wanted? Who cares about predictable schedules? Who cares about delivering for a customer by a point in time which was promised to the customer?
This isn't the way to promote a healthy culture in the company. Nobody should get immediate and total veto rights. In the real world, healthy organizations balance between different stakeholders, and once a path is set, everybody shuts up and executes. The right place to make a stand for your interests is in the stakeholder meeting, not the deployment pipeline.
InfoSec is not optional.
InfoSec is critically important, but it's important just like IT people, janitors and server maintainers - business breaks without them, but they aren't actually earning money and prioritizing them over core business is the tail wagging the dog.
(And yes, I've seen way too much entitled "InfoSec" experts explicitly undermining their own company because they forgot that. Read The Phoenix Project or similar for concrete examples.)
Uhm, InfoSec helps prevent your company from hemorrhaging money and trust in the form of fines and lawsuits. That makes them a touch more important than you make them out to be.
The bigger a companies customers are, the more important InfoSec becomes to your "core business", because the certifications and security required by those customers have large infosec requirements.
How to lose certification: Don't address a known vulnerability (CVE) within a specific number of days, based on severity. Doesn't matter if it's log4j or some random executable in your images that's never used.
When you're up to hundreds of services, thousands of packages, and millions in revenue from Fedramp customers, InfoSec gets pretty important.
There's only ~250 companies / products that are Authorized at any level of FedRAMP, and many of them are explicitly "Federal" versions of their products in order to isolate the organizational controls away from affecting their commercial offerings.
This workflow ensures any work for minor version upgrade is part of the development workflow. PMs can't deprioritise this work against more exciting shiny product features. If you include this as separate development tickets, it just tends to get deprioritised until it becomes a big chunk of work.
Instead of your maintenance work being visible, you're hiding it. You're describing an adversarial culture, one where you need to do shadow work because you don't trust management to prioritize necessary maintenance, perhaps one where management doesn't trust engineering to place maintenance in its proper context. I don't think this is an ideal, I think this is sad.
At that time, you REALLY don't want to find out that you forgot to pin your dependencies and that your older build now doesn't run due to breaking changes and you have no frigging idea what versions of that random NPM crap library were you running 5 weeks ago.
Just... don't do that to yourself. Ever. Pin dependencies, update them regulary, but update them as YOUR decision under YOUR CONTROL. Not under some techbros control based on his GitHub pushes. If you're smart, also host your own repository (Maven, Npm, Docket, whatnot) with your dependencies so your business can't be distrupted by some techbro drama and missing packages.
(This is one of those lessons old neckbears learned on their own skin.)
The most telling part of this screed is that you don’t appear to know you can tell exactly what version you were using at any version-controlled point in time by looking at the lock file for NPM.
Renovate is smart enough to understand that a task such as “Update Node to v16” means updating .nvmrc files, updating FROM in Dockerfiles and engines in package.json in a single PR it creates for you.
But if you want full coverage, there will be a need to write a few regex managers. And testing can only be done in production on the mainline branch. The best way I found to do that was forking the repo to configure and try out renovate separately.
If you want to use some provided official image for vendored software, just port it over.
Then you tie the build+push into your pipeline however suits your org, be it fully manual or end-to-end CI with regular rebuilds.
This solves several issues mentioned elsewhere in the thread. As a bonus you won't be affected should docker.io or quay have issues or make breaking changes.
We now have a basic rule that :latest doesn't make it past proof of concept stage.
This has made automation of our package & container maintenance crucial, since we are a lean shop. It's pretty much a weekly occurrence that a low-medium vulnerability pops up somewhere once your application is large enough.
Maybe we should even be pinning to a specific layer/checksum like so:
FROM ubuntu@sha256:8ae9bafbb64f63a50caab98fd3a5e37b3eb837a3e0780b78e5218e63193961f9> In reality, you probably need a mechanism to enforce tag immutability: e.g., your own registry mirror, or referring to images by SHA).
It's a problem in frontend builds because js libs are quite poorly behaved so we fix more there but any backend stuff, be it alpine / debian base images, jvm libs, toolchains so on we've not had an issue with specifying only major versions.
Dogmatic rules are wrong 100% of the time.
You needed your article to state an example where latest dependency would kill a puppy.
Then your title is valid.
I think it's best to use latest because the combinatorics of the ecosystem as a whole is much smaller. With everything on latest, you have to play whack-mole on individual services to get them to catch up. But you are doing this just when the community at large is most invested in understanding the process of upgrading.
The alternative is that ancient binary A depends on ancient binary B's legacy behaviour, and B has no need to upgrade but A have a very serious need and now you have a serious problem. Maybe upgrading B breaks C for instance. It's almost uncoordinateable at that point once versions drift significantly. You get true technical debt.
Prod stuff....yeah best to pin it for supervised upgrades
I had a CloudBuild fail to deploy due to a bug in one of the builders :latest. Ironically the bug was due to a change in error logging, so it failed silently.
So it's not just docker or dependencies.
In this case I could've used a GCP notify that the builder failed.
Ah, that sounds so trivial. And yet it is very much not. Aside from the hot mess that is the docker registry daemon, some tags are meant to be rolling. You absolutely do not want them cached.
:lts -- newest long term stable release
:release -- newest regular stable release
:x.y.z -- supported but you need find out the exact version each time!There are aspects where reproducibility is key, but thats only a subset of all containers on our network. And on my home network it’s none at all.
This does mean that you have to actively manage your versions for any security fixes. Which you should be doing anyway. However when upgrading or patching you have to check in a bunch of places to make sure the update was effective.
As someone mentioned else where, its better to pin to major versions rather than specific. You'll need to make sure you have decent integration tests and monitoring though. I mean you have that already right?
So it’s like master-SNAPSHOT when we need it to be, but still with reproducibility and regular updates.
As long as you are able to look up what latest evaluated to during the build you retain most of the benefits while getting automatic updates.
latest is useful for builds where you want to know when something will break. If you only stick to pinned versions, it's common for nobody to ever update the pinned versions for years. Then you're way behind the latest versions and suddenly upgrading becomes a huge pain. You may be stuck in a situation where you're forced to upgrade because of a security hole in your pinned version, but to upgrade to a patched version breaks everything.
".....But that's terrible!", you say. "My build won't be repeatable! I won't be able to perform a roll-back build! My build will constantly break!"
All of that is true, but there are workarounds. For example, if you download, version, and store all artifacts used for each build, you can reuse them later. This isn't hard if you take the time to write some scripts. You can pull a container and export it to a tarball and store it. System packages and language dependencies can be downloaded similarly, and repos mirrored. You can do this once a week and version all artifacts with a datestamp, and make your own app builds pinned to a particular datestamp.
If you always build from those versioned archived datestamps, you can always rebuild or revert to an old working latest build.
As far as builds breaking, they certainly will! You need good testing to catch bugs and regressions. But would you rather learn to adapt quickly to broken builds, or have a sleeping tiger waiting to bite you the one time you finally have to upgrade quickly?
Using a stable branch/tag is the safest hedge against frequently breaking builds while still getting security patches. But stable branches still introduce problems. You will eventually need to revert a stable change, and eventually the stable branch will be End Of Life. So even if you use stable, you should still use the practice of installing from downloaded versioned artifacts.
Since EOL will come eventually, you also have to commit to upgrading to a new stable branch and its breaking changes. You must therefore plan to sunset your own code. Find out when your dependencies will EOL, and plan to completely rebuild your apps using latest before then. It has to be a real commitment and not just a "nice to have", because you will end up being forced to do it by a security vuln or repos that stop carrying an old branch.
Ultimately you need to decide how much risk to take and how much planning is needed to avoid sticky situations. Be aware of the consequences of your design and have contingency plans.
Of course, PyInstaller can solve dependency distribution issues - but I've sometimes ran into packages that don't play well with PyInstaller (uwsgi, for example, because it's basically a C program distributed using pip, so PyInstaller can't figure out how to bundle it).
Lockfiles (poetry.lock, Pipfile.lock etc.) solve the survivality problem for the most part.
That's very rare. It's a common trope though and causes amazing examples of startup companies self-destructing themselves when they end up being bogged down in dependency maintenance wars where they're spending a lot of time just fighting all the issues in 100s of alpha libraries they've put on to be "faster". The argument for that was always the one you used. Adding a dependency isn't just "free code", it's also "free code that will probably break at some time and will need to be updated and deconflicted."
In my experience, a good set of older, tested and well maintained core depdenencies is usually much better for company, even if that means that they themselves maintain a piece of functionality that exists as a library.
# GOOD:
image: "nginx:1.21.6"
only full hash reference.
Deletion is possible within limits but it also is with Docker. All the hashes in the world won't bring back a deleted image.
artifacts being removed is much less of a problem as artifacts being spoofed with malicious content.
> This brings up an interesting side point, in that Docker Hub and most other registries allow mutable tags by default. So nginx:1.21.6 might not be the same image today as it was yesterday. In reality, you probably need a mechanism to enforce tag immutability: e.g., your own registry mirror, or referring to images by SHA)