Powering the Python Package Index
caremad.io
caremad.io
If you are on a hiring spree, maybe slip in a job posting for "Infrastructure Integrator" with duties to maintain some projects full time.
It seems like a lot of accounts and effort to manage all these, while AWS, Rackspace and Heroku could host most of the required parts on their own.
Is there a technical reason for using each of these services from these providers? Or is it just easier to get smaller donations from multiple supporters than all from one source?
One of the simplest reasons is as you identified, it's easier to get smaller donations from multiple people than it is to get one large donation from a single company (although we do have large donations too ranging from 30,000/month and going downwards from there-- for all the PSF infra not just PyPI).
Another part of that is a lot of this has grown organically over time and we sought out donations from different providers based on our need at the time.
In addition, I can't think of a single company that actually provides everything we need except for maybe Amazon/AWS.
On top of all of that the more we centralize our donations onto a single company, the more important a single company becomes to PyPI and the larger the amount of Risk we take on is. It would be a lot harder to find a replacement for all of the things we need all at once than it would be to find a replacement for just a single service.
All in all, managing these accounts is not particularly hard (though in some part that's likely because the set of people who has access to any of one of these is pretty static). Most of them provide some sort of standardize API access that doesn't really change based on who is providing said thing (in general, we attempt to rely as much as possible on "Hosted X" where X is some OSS thing we could possibly run ourselves or switch to someone else's "Hosted X" if need be. It's not mandatory but the harder it would be to switch the more we factor that into our decision (for instance, our use of S3 is pretty simple so we don't worry about their proprietary API because it wouldn't be difficult to modify the code to do it differently).
Imagine if Chipmaker X funded some project and wouldn't pay for architecture Y or Z testing or build machines, and when outside providers offered help, Chipmaker X could threaten to remove funding.
I think even the PSF has no full-time staff. Django has 1 person working full time on it.
It's so embarrassing, though honestly I'm doing nothing but complaining about it. While some might argue that it's OK for major frameworks to be like this because they're very stable already, the package manager is a big deal!
Perhaps if someone ran private repos off the same tech for enterprises, and used some of the proceeds to pay somebody to maintain the underlying tech (CircleCI model, basically...)
I think we're making good progress and doing great work, but one of the most surprising points of difficulty in this process has been the acrimonious FUD from the community itself. Various individuals seem to have a perspective that open source must only come from "unaligned" individuals, hacking away in poverty to the detriment of their finances and of their families, and that any "corporate" contribution must be treated with suspicion.
The amount of FUD that we've had to deal with w.r.t conda, Anaconda, etc. is shocking and depressing. There are folks that seem to think it's necessary to fork an already BSD-licensed piece of code simply because it comes from a "corporation", but who are happy to code in Sublime Text on their Macbooks and push code into Github.com.
It bothered me a lot more in the first few years, but I've gotten over it and have learned to just roll my eyes, and keep on keeping on. But for those who wonder why it is that such important open source efforts seem to run on fumes from volunteers (OpenSSL, anyone?), I want to make sure that they understand that it's the community itself that sometimes creates this kind of outcome. Open Source is sometimes a ghetto because certain insiders build walls of asceticism.
For that reason, I almost never directly in https://pypi.python.org.
(For some reason there is vocal opposition to showing any popularity stats, effectively making good packages drowning in the see of not-even-updated ones.)
Edit:
It's not Django, my mistake.
Also, if there are any interested Django developers around, I'm sure they'd love to have some help finishing the project off.
- no popularity markers (downloads, stars on GitHub or both)
- no "search as you type"
- way too sparse (not a good feature for searching)
I gave link to a a search with I consider "the best" (for Sublime Text Package Manager).
Not much of a Django dev myself (I'm using Python mostly for data science).
https://pypi.io/ https://github.com/pypa/warehouse https://warehouse.readthedocs.io/
Also, from the article: "Over the years many different people and services have participated in the maintenance and running of PyPI. I’m not going to attempt to create an exhaustive list of every person or service who has ever helped, but instead focus on those who are currently involved."
However, wherever we do need to spend money, that funding comes from the PSF so the funding comes from donations to the PSF as well as the income generated by events like PyCon.