Are Package Registries Holding Open-Source Hostage?
about.scarf.sh
about.scarf.sh
Putting it all on the registries to come up with a viable business model and provide this to maintainers without any responsibility[1] on the part of the maintainer seems really one-sided.
It costs quite a bit of money to run something like Docker Hub or NPM. If you want something aligned software maintainers first and foremost, you want a non-profit / foundation that's got priorities aligned with the larger community and not a for-profit entity that has to justify keeping the lights on.
Kinda silly headline, too. There are many package registries, but we only see two here that have business models interfering with distribution of software. Only one that's really impeding the ability to host software elsewhere if you don't like their business model.
Docker Hub's rate limits seem unlikely to impact most usage of Docker, and people who're pulling 200 images every six hours should either seek to set up their own registry to take the load off Docker Hub or throw some money to help shoulder the costs. Even if the user's only grabbing Alpine images at 5MB per image, 200 in six hours starts to add up!
[1] Granted maintainers may do a lot of work in actually maintaining the software.
This is an interesting thought. Linux Foundation has a good corporate backing. FSF traditionally provided the backbone in terms of compilers and userland basics. Maybe their 21st century task (and what keeps the relevant in this age) should be such infrastructure.
Apache Foundation is also an interesting candidate for this. I think they also had a good corporate backing.
Maybe there's something technically wrong with the Docker model?
Like CloudFlare? Freemium can be a very successful business model.
Now that they have it, the cost they are willing to pay is based in part on cost-of-switching. Which is pretty enormous, directly and indirectly, when entire ecosystems based on docker have been iterated.
NPM is the odd one out, really. I don't think letting one company control a language ecosystem's single package registry is a great idea, for all the reasons that the author notes!
Seriously though, Fastly's donation of their CDN service is generous and eases the burden on the PSF, but if push came to shove they could definitely afford the bandwidth. In 2018 they had a net income of half a million.
For example CDN77, would start you off at 0.016/gb if you have more than 100TB per month, without any negotiation.
If you have 300TB per day, you surely can negotiate sub-cent pricing somewhere.
No, the costs are manageable.
I think the charitable/community projects setup for developers, need to act like non-profits in other sectors and actively seek out donations.
If PyPI didn't have sponsor providing bandwidth, I'd guess it would implement some form of rate limiting and encourage people to mirror/cache packages much more to reduce load. I don't think it would die completely, but it would be less convenient and still cost the PSF a fair bit of money.
You are right that this donation does not show up in our tax filings because they provide it to any OSS project.
There's absolutely no way the PSF could afford PyPI's bandwidth out of pocket. Last I checked our "bill" from Fastly would be close to $1.5M/month.
Also given that PyPI is critical infrastructure for millions of people and software projects, anything cheaper would not really cut it.
"If you don't support us, we might accidentally forget to audit our packages and feed malware into your build pipelines. It'd truly be a shame..."
In my opinion, every open source project using a foreign-owned domain as their main distribution method is just naive. Of course you'll never be able to keep things constant, because you never had any power over that domain.
As a responsible developer your use case is likely: "I want to install package X, and know that my customers are receiving and installing that package when they perform a build"
The fact that individuals cannot self-host content has been held back by the limitations of DNS (content addressing) and IPv4 (routability) for a long time now.
What I ask is this: if you as a developer were able to self-host the libraries and applications you offer to others -- regardless of whether they're open source or proprietary -- would that not solve most of these perverse incentive situations?
- The bandwidth costs would be yours, but that would allow you to find a charging model that works
- If your software became extremely successful - beyond your own ability to pay for the bandwidth - then the companies and individuals who rely upon your software would be incentivized to step in to foot the bill
- Data about software adoption and usage would be de-rigueur shared with the providers who offer the bandwidth for it
- We would not be reliant (in a community sense, but also in a day-to-day operations sense) on the benevolent albeit often-loss-leading hosting of centralized repositories
Hell I don't know how I would self-host, I just (reluctantly) put stuff on paid-for servers. I guess you start by calling your ISP and asking if they can pretty please give you a static IPV4?
Even still, you don't need a static. Easy to do DNS map and punch a hole in your FW or have some DMZ.
There is not a way to solve for zero-cash and zero-work
Smartphones and data plans would provide the bare necessities for self-hosting; in many cases there's abundant computing and bandwidth resource available to them. Many packages/containers are small and downloaded infrequently - any much data capacity, I expect, goes unused.
If and when smartphone-hosted resources become insufficient, a cloud VPS like your approach would be a next logical upgrade. Beyond that, corporate/foundation-based sponsorship and dedicated servers and bandwidth.
The key would be to make it near-seamless to migrate between those different environments. Namespaced source code repositories and packages appear to have worked well for the likes of GitHub, GitLab, NPM and Docker Hub, so perhaps following similar conceptual design ideas would make sense.
A few areas of concern would be:
- How do you keep end-user devices safe if they will be hosting content for a wide audience?
- How do you react to credentials and other protected content being posted if repositories themselves are a distributed network?
- How do you achieve discoverability and search of content in a distributed environment?
... not to mention whether the effort and migration to such a model is worth the benefits.
I tend to think it would be, since it aligns the incentives around spending, increases hardware utilization, and increases resilience by removing single points of failure.
That said, I also imagine there are well-founded and sincere arguments for continued centralized code and container hosting that are valid and worthwhile (not least of which: it's where we are, and it's relatively straightforward to reason about).
[1] - https://blog.printf.net/articles/2015/05/29/announcing-gitto...
[2] - https://hn.algolia.com/?q=GitTorrent
[3] - https://radicle.xyz/
The problem with Docker Hub's pricing model is there are actually two use cases for Docker Hub. One is where some random entity makes some software -- they don't have any money, so it makes sense for the user to pay to download it. It's cheaper than setting up your own CI and hosting to build images, after all. That's the only pricing model that Docker Hub supports right now. It may be annoying that it used to be free, but that was just an accident -- you should have been paying for Docker Hub pulls from day one.
The other model is where some commercial entity wants to distribute software to their users. In that case, they'd be happy to pay for their users to anonymously download it. But, Docker Hub doesn't support this particular model, and that's what's causing a lot of problems. (Where I work, this is already a problem for our customers. I think we're going to move to GCR, and like the article mentions, this is a pain because we are locked in -- you can't make Docker Hub 302 to your new host. Everyone has to rewrite their configuration to pick up the new registry, all because Docker won't let us pay for their pulls!)
The article also talks about NPM being problematic. I kind of agree with this; I spend a lot of time waiting for my CI system to re-pull node modules. (Because everything is done in containers, nothing is preserved between builds. Things can be preserved if you want them to be, but on CircleCI it's actually faster to re-pull all the modules from the Internet than to restore from their own caching mechanism. Shrug.) I had this problem with Go when modules first came out (our builds were getting ratelimited by Github because a lot of our dependencies happened to be hosted there), but it was easy enough to set up a local module proxy, and it was never a problem for us again. (Go has since decided to run their own module proxy, and I stopped running my own. Theirs is great! But obviously, someday it won't be free anymore, but at least running your own is a first-class option.) NPM, it seems, have never really offered this as an option -- they've never rate limited me, and their documentation says "don't worry about it!" which of course makes me worry more ;)
This has hit us big-time last week -- all our k8s components are hosted on DockerHub and our firm is happy to do publisher-paid pulls, but it seems to not be an option. Instead, every customer is expected to get a paid account (if they pass some download threshold, which they may or may not.)
What are others doing? We're thinking of moving to Azure Container Registry or Amazon Elastic Container Registry. We're happy to pay just to ensure customers are not randomly throttled, though I wish we could just keep it all in DockerHub (we'd be fine paying DockerHub for customer usage, but we cant force customers to do so themselves.)
Most, I wrote, myself, maintain on GitHub, and include as Swift Package Manager dependencies. I write in a modular, layered fashion, and try to fork off as many components as possible into standalone projects.
I'm very, very careful about including third-party dependencies. I think that these are the only ones that I use, throughout my projects:
SOAPEngine (Paid)
ffmpeg
VLCLib
SwiftKeychain
I may have one more, somewhere, but I can't remember, and I'm too lazy to look. We can rest assured that it was not lightly added.
It's easy to complain, it's more difficult to work on solutions. We should all be doing more of the latter (working on solutions).
Personally, I'd much rather see a solution to a problem than a complaint. The solution might involve discussions that go back and forth, but if they culminate in a decision on a way forward with a person willing to do the work, they are much more useful than "your code sucks on line".
Complaining isn't contributing.
Edit: also, working on a solution doesn't preclude discussion on the proper solution.
I too was initially surprised but then I thought of github (this was also the case pre-microsoft acquisition), sourceforge, gitlab, etc who all seem fine with me downloading/cloning as many repos as I want without charge.
Artifactory seems to have a pretty big chunk of this vertical. It supports a few different repository protocols, so it serves as a bit of a one-stop shop that survives technology changes.
As you get into CI/CD you start to notice that your upstream repo is occasionally down, because it’s getting in the way of some deadline.
Even ignoring download costs, a local cache (one of the functions of an artifactory) helps speed up those downloads and,with it, your builds. It probably also helps against getting blacklisted by code repositories.
An artifactory also automatically backs up any libraries you use. That protects against them disappearing from the internet.
I guess you could still fall back to the main package source if the local mirror is down.
1) set up a series of N docker registry mirrors in pull-through mode (https://docs.docker.com/registry/recipes/mirror/, it's as simple as "docker run --rm --name registry -d -p 5000:5000 -e REGISTRY_STORAGE_DELETE_ENABLED=true -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io -v /mnt/persistentdata/registry:/var/lib/registry registry")
2) expose them on the same domain name (multiple A records, loadbalancers, whatever you want)
3) set them as mirrors in each machine's docker daemon
In case one of your mirrors go down, take them out of the DNS/LB rotation. That's it.
There'd need to be a way within docker to alias to the new URL so that what normally would go to docker hub ends up pulling from the mirror.
The only downside is, as said, that it can't cache third party repos (quay.io comes to mind for people involved in k8s). For these, one has to mess with the resolv.conf and self-signed HTTPS certs for the Docker registry mirror.
It runs well, is easy to keep up and working and generally was awesome.
One objection that I'm sure will come up is "what if people stop hosting a package you depend on":
That's where dedicated package hosting services (like npm) can come in and provide a reliable source of these packages that's fast and always available (potentially for a price). The benefit over the status quo is those services will be commoditized, so they have to compete purely on price/reliability, since as a user you don't have to care where the packages come from in a content-addressed system.
Anyone can host their golang package at example.com/myPkg and Go wouldn't care less what/who runs example.com
> ..npm, and other comparable registries are incentivized to create lock-in
The author’s conclusion doesn’t match the actions of NPM. There are open source implementations of both client and server, you can run your own NPM registry, and you can change the default registry easily.
> What makes npm’s particular scenario even worse is that they've made it so difficult to use a registry that is not npm
Uh?! I’m using Verdaccio (an open source Npm registry) everyday, the setup is extremely easy. Also yarn uses their own registry by default.
I don’t see where is the “lock in” in NPM.
Edit: typo
https://github.com/scarf-sh/scarf-js#as-a-user-of-a-package-...
Perhaps a balance could be achieved by rate limiting by IP to encourage caching by devs. I'm guessing a fair amount of waste occurs from CI testing. Normally once an end user has the image that should be enough.