If you do something like this, you absolutely MUST have a local registry.
Harbor [1], JFrog [2], and Quay [3] would be the first ones that I look at.
Harbor is open source, free, and a member of the CNCF. You will need to do a little bit of work to set it up to scale properly. JFrog offers a SaaS registry, but you will pay big $$ based on pull traffic. Their commercial site license is about $3k/year. Quay is older than either of them, stable, and high quality. I'd start with Harbor these days.
[1] https://goharbor.io/ [2] https://www.jfrog.com/confluence/display/JFROG/JFrog+Artifac... [3] https://quay.io/
I have pulled and run 20k times a 1GB image in less than 10-15 minutes without breaking a sweat.
Finally GitHub packages offers a registry out of the box . It is great for CI and devs to access . I generally have the tags mirrored from tags GitHub for production to ACR .
1) It is broken and unusable on Kubernetes and Docker Swarm.
2) It is flaky often returning 500 type errors.
3) It is expensive as the amount of pull bandwidth is very limited.
Hmm, I use them on several kubernetes clusters in the past few months and don't see any issue yet.
The main issue was ECR has a slightly different authentication model than docker swarm. The whole '--with-registry-auth' only partially works when you are using ECR. Unfortunately, it works just enough that you think it's working, until all your tokens time out and a worker can suddenly no longer pull an image.
Our common failure case was an image becoming unhealthy or a node being drained. When that image would try to be restarted on a different worker, if that worker did not have the image it would try to get it from the registry. If the tokens were expired it would fail.
The only "fix" we ever found was to setup a cron job that forcibly deployed a new version of a "replicated globally" image every X minutes (where X was based on ECR token expiration). It kind of worked, but we still had occasional failures we could not identify.
I wish it worked better, because it was nice to use ECR. Frankly token expiration sounds much more secure too, but without direct support for token refresh inside the docker engine it's just hard to get everything to work
That said, word of warning for anyone looking at GitHub Packages for docker registry: it's broken with containerd and some other similar tools. They (GitHub) are currently working on a fix: https://github.com/containerd/containerd/issues/3291
It's not transparent though.
With ECR you pay for image storage: $0.09 per GB after the first 1 GB which is free
pull from docker hub once, push to ECR. then pull from ECR as much as wish
you can set it up in less 10 min and the only thing required is to add '--insecure-registry' in your client. It is not a issue if all your machine are in private network.
still wonder how to do it in minutes.
Expect to spend 1-2 hours first time you try it until you can setup the correct DNS records, API keys and configuration.
Afterwards it's pretty hands off, every three months you'll receive an email from letsencrypt and you'll have to rerun this script to regenerate your certificates. Takes 2-3 minutes max (but of course you still need to distribute your certificates to all relevant services...)