Hacker News Reader

Comment by digisign | Hacker News Reader

We had mirrors on every continent for linux distributions (and other things) way back before the turn of the century! cough cough

Mildly surprised a service as big as PyPI doesn't.

View on HN

I think the idea here isn't that there isn't a PyPI mirror in Africa. It is that not everyone has internet, so this person wants to have a tiny computer output a local WiFi network people can connect to and download pip packages. Imagine a small town or village with some power and a classroom, but no internet. A teacher could setup a network and have students connect to this device and download packages so they can complete some assignment/make the next Facebook.

Or that the Internet access they do have is often metered. A friend tells me 100MB costs about 1 USD where he is, which is not an insignificant amount of money. Really puts the whole 300 MB electron app thing in perspective; at any rate it's understandable why having a PyPI mirror in the classroom would be preferable to having each student download the packages over and over.

> Mildly surprised a service as big as PyPI doesn't.

PyPI uses Fastly as a CDN, which does indeed have presence on every continent (except the big, cold one). The problem here isn't presence, but connectivity to the outside Internet itself.

Source: I'm an active developer on PyPI.

I think PyPI is mostly served via Fastly.

https://dustingram.com/articles/2021/04/14/powering-the-pyth...

I remember at pycon ~2016 one of the maintainers of pypi did a short talk on it and the entire pypi service at the time only ran on one or two boxes. It was surprisingly scrappy for such a critical service.

This has changed a lot in recent years and is now in a much better state. Their docs give a really nice overview of the evolution over time https://warehouse.pypa.io/application.html

PyPI needs more funding. PyPI even disabled their search API for the pip CLI because of infrastructure overload. It would be nice if more sponsors stepped up to fund their infrastructure.

I think companies could choose to be more responsible with their usage. Looking at PyPi utilization, I have to imagine the bulk of it comes from CI/CD tooling hammering the servers without any intermediate caching.

Definitely. I don't think hammering PyPI's API would be much of problem if the companies doing the hammering paid for the privilege of doing so, though.

That's definitely true. There's so many build jobs running in docker containers that lack a pip cache. I believe my company has its own package servers, but I've specifically avoided adding build system features because of that lack of caching. For one of me, there's probably a dozen people that would just hit the main pypi servers every run.

Serious question: should charities start downloading the entirety of NPM and PyPi (+ Node, NPM, Linux packages, etc.) onto hard drives and passing them onto schools teaching software development in third-world countries alongside the computers they normally send? I'm not sure if that's even possible, or how big the HDD would have to be to store all that, but it would be very very nice nevertheless. I can't imagine not being able to install one-off software packages.

PyPI is more complex and probably serves way more traffic.

Reply on news.ycombinator.com