What you need to know about upcoming Docker Hub rate limiting
docker.com
docker.com
I have to admit that 100 pulls per 6 hours doesn't sound like enough for even a small team doing CI/CD.
But in the long run, it might make a lot of organizations more lean in how they do builds, which could be a good thing.
In my POV the first fundamental change that this caused was to cease considering docker hub as reliable infrastructure, and therefore started to look for solutions to reliably source container images. For instance, GitLab already provides its own container registry, and thus it makes sense to simply source images from them than from tertiary sources like Docker Hub.
Another impact that this has is that I'm looking into the need to operate a custom container registry for each Kubernetes cluster. It wouldn't be nice if an application running on Kubernetes or Docket Swarm risked downtime just because you happened to hit one of Docker Hub's rate limits.
I do recall reading about the free pulls a year ago and thinking, "Wow, that's awfully generous!" :)
We don't need to resort to paid services. This is all about convenience.
As I've said, services like GitLab already provide container registry services, and we are also free to operate our own custom container registry services. Docker Hub is only used to source reference images such as Ubuntu, postgres, rabbitmq, traefik, etc etc etc. We can simply point to a different provider to continue accessing those. Even if these projects continue to use Docker Hub then if we stick with stable releases then we can simply point to a different container registry service, even one managed by us.
The main point is that Docker Hub ceased to be a reliable provider. Thus, we plan accordingly.
Maybe we understand something different by a "small" team, but IMHO a smaller team uses more ready public images, to save time and costs. Also even not that big project on Kubernetes that includes internal software will also have a lot of other software.
Let's speculate: I had a look at some teams I know and how they work (or they would like to work soon) and it's easily 15-30 images in a CI/CD pipeline. Plus tools used to develop, but not exactly released in a build (source code management, PM, communication etc). All teams I know have 2-3 cycles developed at all times (next release, hotfix, future release after the next one). This means a lot of these tools are multiplied by 2-3 (upgrading tools as well, so multiple versions of same tool needed). If you have a few clusters that have different requirements it can also be multiplied by 2 or 3.
I would not dare to think what would happen if someone used "latest" tag in their daily work, or organized a docker/kubernetes workshop for new people :)
It really depends on your usecase. I mean, each CICD stage can involve building a brand new image and running a couple of them, sometimes more. 100 pulls per 6 hours corresponds to a constant rate of 3 min between pulls, which would require between 15 to 20 minutes for each commit.
Anyway, the primary practical implications of this sort of change is a) rate limiting suddenly becomes a requirement, b) we can no longer count on Docker Hub to provide a reliable service, thus we plan accordingly.
I may be wrong but I interpreted that to mean that even caching won't prevent you from hitting the limit.
Just a few years back I remember even the smallest Java shop would have a local "caching/forwarding" Maven repository and the upstream repos were only considered a fallback.
Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit.
A limit of 100 pulls per 6 hours averages to a pull each 3 minutes. Each stage of a CICD pipeline uses more than 1 container image, and it's customary to have CICD stages build and pull from previous stages, such as integration tests,performance tests, canary tests, etc.
I would dare say that you're doing something terribly wrong if you don't need to pull images often, because that largely tells you're doing barely any tests or deployments.
Yes.
> If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull.
Yes.
> Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?).
Not necessarily. You dockerize an app prior to running non-unit tests tests, and that image needs to be stored by a container registration service.
With this change, Docker Hub's ceases to be that service.
> You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds.
Nope, the point is that Docker Hub ceases to be relevant as a container registry. As long as you start to do all your work outside of Docker Hub, you won't need Docker Hub to do any of the work. You just get your base images from where you store your own container images and suddenly Docker Hub stops playing any role in your workflow, thus losing al relevance.
If you're in a situation where having your own registry means Docker Hub "loses all relevance", then you're probably not the kind of user they have any reason to care about (and that's nonsense anyway, because you'll still almost certainly be using 3rd party images from there).
I'm sorry to say but your insistence of speaking out of a position of blisful ignorance and your refusal to open your eyes does not change the world around you.
> but I've always seen Docker Hub as...well...a hub, not just any ol' registry.
No one with the faintest cursory knowledge of Docker would ever claim that Docker Hub is anything other than a docker registry.
Docker itself advertises Docker Hub as a service "to create, manage, and deliver your teams' container applications." Docker itself describes Docker Hub as a "cloud-based application registry and development team collaboration services." That's their own official marketing slogan, taken straight out of their "Why Docker" page.
And even that you managed to miss and failed to understand.
> It's where people publish final builds
It really isn't. That's not how containers work at all, and that's not how Docker promotes the service, and that's not how anyone uses it. Nothing in your comments is even remotely compatible with basic notions of CICD.
Apparently you're so out of touch with containerization and CICD and devops workflow that you're even entirely ignorant of basic concepts like what a latest tag is supposed to represent and how tags are used to update apps.
The fact that you try to pass off that absurd assertion in such absolute terms makes it quite clear that you have absolutely no idea about what you're talking about.
Hell, your claim can only possibly be made by someone who never even registered a user account in Docker Hub. Please refrain from speaking about stuff you know nothing about.
> If you're using it for basically passing your own images between CI workers or whatever other workload that requires you to constantly be pushing and pulling
This mindless statement just makes it clear as water that you have no idea of how containers are used. At all. And yet here you are degrading the signal/noise ratio by blabbering about stuff you know nothing about.
Either way, I can't be bothered with this thread anymore as there's clearly no more constructive discussion to be had with you. Cheers
Most CI systems poll to see if there are new versions of an image. If this is done with a GET request (to get the manifest as a first step towards pulling layers), it counts towards the rate limiting.
> Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit.
Maven separates the domain of a Maven repository from the coordinates of packages. Docker does not. Dockerhub's domain is inferred into any shorthand reference like `foo/bar`, meaning that it will ignore a pull-through cache.
You can ostensibly configure a docker daemon to treat another registry as a trusted mirror, but it's harder than it looks in practice, because on most production systems that won't (and shouldn't) be enabled.
EDIT: Yeah, that makes sense given the protocol's design. Anyways, HEAD requests don't count, so hopefully the client can utilise those (or be modified to). [/EDIT]
And yes, Maven has a design much better suited to having a local "proxy" (in fact, it always does that on localhost anyways), but the setup looks pretty straightforward and I really don't see much reason to not use it on all dev, test and CI machines. I'd also use it in production, but even if you'd prefer not to, production servers don't pull that often and updates are usually done through hooks, not constantly polling a repo.
So basically, from my perspective (handful of servers, each with some 20 containers) this change will be thoroughly annoying, but will force me to set up caching which will make things much faster and also decrease the reliance on an external service for day-to-day operation. Not a fan of it, but yi completely see the reason why they did it.