Hacker News Reader

It’s not common to need that many pulls nor is it hard to build your own images.

If you’re deploying to a cluster with 200 machines, you could easily hit this if you use the public registry though. However, if you’re managing that size cluster you can probably afford the fee, but more importantly, you should probably pull once to a local registry and use that to deploy to your cluster anyway.

Do you run a local registry? Any high-quality articles/youtube talks to share? I'm about to set one up for our own little cluster (~5 machines, ~75 containers). I know tons about docker engine, and a fair bit about the registry, but it's always nice to watch a "lessons learned from actually doing this in production" talk to know what mistakes to avoid

If you want to run a local registry to stay below the 100 pulls per 6 hours limit please consider GitLab. The Dependency Proxy https://docs.gitlab.com/ee/user/packages/dependency_proxy/ will cache docker images. This way you stay within the limits Docker set and subsequent pulls should be faster as well.

I run tens of thousands of docker images in production, or rather, tens of thousands of copies of a few hundred images.

If you do something like this, you absolutely MUST have a local registry.

Harbor [1], JFrog [2], and Quay [3] would be the first ones that I look at.

Harbor is open source, free, and a member of the CNCF. You will need to do a little bit of work to set it up to scale properly. JFrog offers a SaaS registry, but you will pay big $$ based on pull traffic. Their commercial site license is about $3k/year. Quay is older than either of them, stable, and high quality. I'd start with Harbor these days.

[1] https://goharbor.io/ [2] https://www.jfrog.com/confluence/display/JFROG/JFrog+Artifac... [3] https://quay.io/

I was a happy user of JFrog's registries via site license at my last 2 places. Seemed to just work as expected. Didn't have visibility into the cost though (other teams set it up) so I had no idea it was $3k/year.

Just to add all the major cloud service providers provide registries ACR /ECR/GCR etc . If you run k8s service with one of the them in my experience it is best to use the corresponding registry.

I have pulled and run 20k times a 1GB image in less than 10-15 minutes without breaking a sweat.

Finally GitHub packages offers a registry out of the box . It is great for CI and devs to access . I generally have the tags mirrored from tags GitHub for production to ACR .

Looking at doing GitHub Packages for direct-to-dev and mirroring into ECR over here. Seems sound. But also considering other options as ECR is a pain to work with.

That said, word of warning for anyone looking at GitHub Packages for docker registry: it's broken with containerd and some other similar tools. They (GitHub) are currently working on a fix: https://github.com/containerd/containerd/issues/3291

I got set up with ECR without any difficulties whatsoever. You do have to authenticate before pulls and pushes, but that can be scripted very easily.

Github Docker Registry is a mess and should be avoided at all costs.

1) It is broken and unusable on Kubernetes and Docker Swarm.

2) It is flaky often returning 500 type errors.

3) It is expensive as the amount of pull bandwidth is very limited.

> It is broken and unusable on Kubernetes and Docker Swarm.

Hmm, I use them on several kubernetes clusters in the past few months and don't see any issue yet.

Github packages works with Github CI out of the box, it makes development lot easier, like I mentioned for best networking in prod you should always use the registry from your k8s Provider, mirroring the Github registry to ECR/GCR/ACR is fairly straightforward. Bandwidth costs are eliminated, network is lot more reliable intra DC.

FYI, Using ECR with Docker Swarm is something we did try. It was hellish. We never nailed down the exact problems, but we spent about a month with 2-3 experienced engineers trying to fix the edge case issues.

The main issue was ECR has a slightly different authentication model than docker swarm. The whole '--with-registry-auth' only partially works when you are using ECR. Unfortunately, it works just enough that you think it's working, until all your tokens time out and a worker can suddenly no longer pull an image.

Our common failure case was an image becoming unhealthy or a node being drained. When that image would try to be restarted on a different worker, if that worker did not have the image it would try to get it from the registry. If the tokens were expired it would fail.

The only "fix" we ever found was to setup a cron job that forcibly deployed a new version of a "replicated globally" image every X minutes (where X was based on ECR token expiration). It kind of worked, but we still had occasional failures we could not identify.

I wish it worked better, because it was nice to use ECR. Frankly token expiration sounds much more secure too, but without direct support for token refresh inside the docker engine it's just hard to get everything to work

We have not had good luck with Quay. They are not stable, especially as of late. There was a period last month where for two weeks pulling images was a crapshoot.

Thank you very much. This is exactly the type of info I needed.

Just drop a Sonatype Nexus instance on a Docker container somewhere on your network. Alternatively, use Squid if you don't push to the public Docker registry, although you might need to mess around with internal CA for SSL...

Nexus in a container... because storage in containers is such a good idea? Any vps with a disk is probably a better idea

You can still bind mount a directory into a container...

Storage in containers has been a long solved issue. The defaults are unfortunate because but make sense for ease of use. Your container root should be read only, ephemeral storage lives in a tmpfs or dynamic volumes depending on performance and size needs, and persistent storage lives in volumes.

I would stay away from Nexus. It has problems with latest tags.

Docker supports proxies (they call them “pullthrough repos”) so you don’t have to be so generic as an http proxy.

I just use ECR[1] which in many cases costs less and is fully locked down behind my AWS VPC

With ECR you pay for image storage: $0.09 per GB after the first 1 GB which is free

[1] https://aws.amazon.com/ecr/

are you gonna rebuild all the images that you use and push to ECR?

nope. you don't have to rebuild images to push to different registries

pull from docker hub once, push to ECR. then pull from ECR as much as wish

Personally, I wish generic caching proxies were still a thing, and easier to set up. I've tried setting up squid several times in the past, and failed miserably every single time--all I want to do is use it as a gateway (ie, make the proxy invisible to the application) for e.g. apt packages, so I just ended up using apt-cache or whatever other appropriate software, but I'd far rather use something generic that just works on 90% of the software I use at home, whether it's reading webcomics or repeatedly installing the same software in a dozen VMs with slightly different configurations, or even just browsing remote filesystems via webdav.

I use nginx to proxy cache the Arch Linux package repository transparently. It's fairly easy to set up, and enables nice features like contacting a secondary mirror if the first one is down, or when multiple requests hit the same resource, all are blocked waiting for a single merged package download, so the proxy will not make the download multiple times if I run pacman -Syu on my 18 machines in parallel. And it's all just 20-30 lines of nginx config.

It's not transparent though.

this is my way: https://github.com/EnigmaCurry/lazy-distro-mirrors

https://docs.docker.com/registry/

you can set it up in less 10 min and the only thing required is to add '--insecure-registry' in your client. It is not a issue if all your machine are in private network.

Isn't there no authentication on that registry? I guess that's fine if you don't believe in zero-trust architecture.

you are right. That is what you can get in minutes.

If you cannot get a TLS cert for internal infrastructure in a few minutes, I'd recommend you start looking into why.

no good document on it and it is not very important for me ( I run it on homelab).

still wonder how to do it in minutes.

I use this (in a docker image) to generate certificates automatically: https://github.com/adferrand/dnsrobocert

Expect to spend 1-2 hours first time you try it until you can setup the correct DNS records, API keys and configuration.

Afterwards it's pretty hands off, every three months you'll receive an email from letsencrypt and you'll have to rerun this script to regenerate your certificates. Takes 2-3 minutes max (but of course you still need to distribute your certificates to all relevant services...)

If you run traefik it's even easier: https://docs.traefik.io/https/acme/

If you run on Kuberenetes, the image is/can be cached at network layer.

Anything is a deal breaker if the user expects to pay nothing. That is why going from free to $1/month will have a much larger user dropoff than, say, $1 to $10.

Weber's Law. Human perception is logarithmic.

It may be general resistance to a concept. More and more services are becoming subscription based, so 5$ here, 5$ there and the cost creeps further.

Exactly.

It's hard enough to save money as it is.

News websites blow my mind with this - if I forked over $5 to every news outlet I occasionally like to read, I'd be spending at least $500 maybe more per year JUST to get access to some random person's biased recant of what's happening in the world. If there were a news source that did the opposite of this, and basically provided a bullet list of objective, non-biased events boiled down to exactly what I need to know, that might be something I'd pay for. Hell, it would save you time over filtering the opinionated BS out.

Mind blowing, imagine paying $5 for each newspaper one wants to read.

Except it’s not $5, it’s $5 to sign up, then an email and a phone call and your firstborn dog to unsubscribe. If it were microtransactions, that’d be one thing...

If the service doesn't play ball there is always the consumer protection agency.

That doesn’t work. I find it better to stay anonymous and avoid spam from these services.

The perpetual spam is worse for me than the $5.

You’re missing the point. If there’s a news source that you occasionally read then it’s far more cost effective for you to just buy the paper at the stand for 50¢ the few times you want it. Same with magazines. If you read every newspaper then getting lower cost and delivery in return for the paper getting consistent revenue is a good deal for both parties.

If you get your news like most people, via link aggregators like HN, Reddit Facebook, Twitter then you get linked to dozens of publications that all want a $5/mo. commitment which is untenable.

It would be neat if news sites would start offering 50 cent day passes instead of difficult to cancel subscriptions.

I sort of half thought apple news might go that way. might not be cost effective - bundling larger subscription stuff is probably more revenue/profit.

But... in their news app, there's always a couple of interesting articles I might want to read, but I'm not signing up. They have my info, and a Touch ID device I'm holding tied to my payment info. "Read this article for 50c?" I'd certainly give some a read now and then.

My daily newspaper costs 2€ at the newsstand, not cents. With a monthly subscription of 5€. Pretty much worth it.

I have been subscribing valuable information sources since 1995, so I do get the point.

I explored building exactly that but turns out there’s no money in it. Most people want the narrative with the facts, if not more so.

There's a big market for this, it's just not for individuals. For example, Bloomberg distributes factual news on its terminal. The Bloomberg terminal even highlights important words in news stories so you can absorb the information more quickly. So if there was a earthquake somewhere, it might highlight the word "earthquake," the number of people that died, and the economic cost, for example.

Also there are news wire services that do mostly what you're describing. If you just want to be entertained (most people read news for entertainment), then they don't really care about the facts. They want to hear about so and so blasting so and so or whatever. But if you're trying to make money from information (traders, journalists, etc), then you really don't want to be reading the kind of stuff the New York Times is publishing.

I ran the math on this once.

Just a subscription to The Information is $399/yr. Add a subscription to the Times, and you've already blown past your $500 budget.

Providing objective, non-biased events would be very hard.

Consider for example the current riots going on in the US. How do you objectively report on that? With bias, on one side you have "peaceful protest disrupted and escalated by the police", on the other side you have "police intervening in riots to maintain order and protect property". There's not really inbetween.

"Protesters say their peaceful assembly has been disrupted and escalated by the police. The police argue they've only been intervening in riots to maintain order and protect property."

Done.

If you have to represent "both sides" (in many cases there'll be more than two sides really), you end up having to give a voice to nutjobs, plus you present both sides as equally valid assessments.

Much as we'd all love an "unbiased" news source, the reality is that bias is a very hard problem to solve well.

Indeed, just repeating what people say about an event may be factual, but without any concept of what is actually true, it can’t be considered objective.

If one side is lying, objective reporting would tell you which side it was.

$5/month isn't $5/month. It's convincing your boss you need $5/month, because they need to convince their boss, which eventually makes its way up the chain to C-levels, who don't know Docker from yesterday's rotting tuna casserole and view eating either that or the $5/month with the same level of disdain.

It isn't about the money, it's about the Mommy-May-I up and down the chain with emails and meetings and careful explanations to skeptical glares. It's a psychological and institutional barrier.

If your C suite is personally approving a $5/month charge, your organization is likely no where near the size where a change like this from Docker impacts you.

Or maybe your organization is a public institution, where each penny spent needs to be authorized/accounted for.

> It was revealed on January 22, 2009 that Thain spent $1.22 million of corporate funds in early 2008 to renovate two conference rooms, a reception area, and his office, spending $131,000 for area rugs, $68,000 for an antique credenza, $87,000 for guest chairs, $35,115 for a gold-plated commode on legs, and $1,100 for a wastebasket. Thain subsequently apologized for his lapse in judgment, and reimbursed the company in full for the costs.

> https://en.wikipedia.org/wiki/John_Thain

"Sorry I got caught. I will work harder to hide next time."

Even in a public institution that has never failed an audit, I have trouble believing $5/mo is a hard thing to get. Technically, its $60/year fee because its one of those annual plans. We pay for vultr with no drama.

Worked at a place where everything went through the VP finance and they were making $15M/yr profit. At one point we brought ion a desktop off of the curb to use as our build server. Sure, we got in trouble with ops later but those above us didn't care since it didn't cost us anything.

> If your C suite is personally approving a $5/month charge, your organization is likely no where near the size where a change like this from Docker impacts you.

Or has toxic micromanaged structure. I've had friends who have worked places that would barf over ongoing $60/year software charges, where anything like that would have to go up to C levels and require justification. Luckily never worked at one myself, dodged that particular bullet.

My purchase-approval flow is the same for $1 purchases as it is for $1k purchases. If we hadn't finagled a minor workaround, it would be the same as that required for $5k purchases.

Handling each purchase and documenting it in case we are ever audited requires easily $25-50 of people's time.

If your C suite requires approving a $5/month charge the company has serious issues. This always bugs me about the HN attitude about spending money, but it's really a extremely frugal developer complex. It's $5 a month, you get insane value out of Docker Hub, just pay it. I come from an entrepreneur attitude, my time is my most precious commodity. I don't optimize minor expenses, I optimize the big picture, my time, outcomes, and revenue coming in.

Yeah, when it's $5/mo blocking good utilisation of multiple $150k people, someone doesn't understand priorities.

$5 a month also means consideration is given as part of a contract, and legal will need to review to make sure you aren't granting them patent immunity, etc.

It took 3 months at my old client (fortune 100) to get a signature for an Addendum that would deliver us a service for free in addition to the paid stuff that was already signed.

For me it’s not the c-suite, it’s the admins who make purchases. Or the hassle of maintaining a corporate card.

The headaches to spend $5 cost way more than $5.

>It's convincing your boss you need $5/month, because they need to convince their boss, which eventually makes its way up the chain to C-levels, who don't know Docker from yesterday's rotting tuna casserole and view eating either that or the $5/month with the same level of disdain.

this is where miracle of enterprise sales happen - the $5 subscription can be sold as a $50K+ deal by smooth enterprise sales who will provide the C-exec with the experience making him feel like he did something smart and great for the company.

Just promise to give the exec a keynote time slot where he can describe the ROI of the 50k deal. Helpfully, the vendor will provide an Excel doc that lets you calculate the benefits. You don’t even have to realize them, just aspire to. From there, the C-level gets some fawning press coverage for their LinkedIn profile and a set of job offers at the next-size-up organization where they can also level up on comp...

There is no way you need C-level approval to get a $5 a month subscription.

Start looking. That place is going to fail.

If you work for a company that disconnected from reality you best be looking for a new job. $5 the guy who decides budget should be angry that you didn't come with him directly for such a tiny amount of money.

This "up-and-down chain" should stop pretty quickly at the level where eng mgmt knows loaded costs. It's an eng manager and director's job to compute total-cost-to-execute, and opportunity cost (using loaded-costs for engineer time). They should be able to approve once you've convinced them that it's cheaper to buy than to build.

Engineers can head this off by prepping a total-cost-to-execute analysis for mgmt. This doesn't need to be complicated. It's just some estimates of what's needed and why, and what the alternatives cost. My eng VP used to ask me for these when I'd send up a request. He wanted to know that we thought about total-cost-to-execute. He'd usually only read the exec summary and approve. If these requests are really going that far up the mgmt chain either someone isn't doing their job, or higher level mgmt are micromanagers.

>Now my question has changed to: is $5 a month really a deal breaker for folks to get unlimited?

I want to speed forward 5 years and see how well this ages. It reminds me of all the other comments about "of course Facebook will never require an account to use your VR headset"...

I think it will age fine. If every ad-supported website switched to this model, I'd be paying for either zero or one websites.

If it was $5 for access that would be a completely different situation, but a big free tier followed by $5 for unlimited is fine.

I don’t think anyone who’s adopted systemd can call themselves a curmudgeon :)

Sometimes, curmudgeonliness is measured in resistance to changing the default:

"Stock CentOS was good enough before and it's good enough now!"

It is pretty cheap, it is just the free beer crowd that is upset.

Yeah, I'm just going to pay for it. I think I probably already am, but if I'm not, I will. Even the enterprise 7/user isn't that bad at all. Github enterprise is like 3x that.

The limit for unauthenticated users is by IP address. I could imagine a smallish business that has a consistent on-ramp IP for their users that could breach the limit not on any single user but in aggregate.

I do sympathize with Docker though: Storage and bandwidth at that scale isn’t cheap and they need to monetize somehow.

You should be more shocked that everything else is so expensive.

The base costs for a lot of tech stuff (like bandwidth) are so cheap. But you would never know between these bullshit "what is it worth to you" pricing models and the number of middlemen trying to stick their hand into the pot. It's disgusting.

Developers are cheap. “I could build that” is your competitor

Note could, not will

Developers are overly optimistic, undervalue their time (they don't know the loaded-cost of their time), and overvalue their solutions (they rarely account for maintenance costs or the long-tail of time to perfect the solution). So developers think they are cheap. For example, I had an engineer express disgust that we might have to pay $4/month-per-user for gitlab just to have mirroring enabled. He said "we can do our own mirroring." When asked how long it would take he said, "between 2 hours to two-weeks if we have to iron out bugs". So when accounting for his time, his price to deliver ranged between $200-ish in the best-case scenario, to $8000-ish in the worst-case scenario (which far exceeds the cost of paying for the licenses). Needless to say, we paid for the licenses. He also wasn't considering the opportunity-cost, i.e., what will he NOT do that's more valuable while he works on this proposed solution that we could just pay for?