One of the big companies making billions on Python software should step up and fund the infrastructure needed to enable PyPI package search via the CLI, like you could with `pip search` in the past.
(I think the biggest blocker on CLI search isn’t infrastructure, but that there’s no clear agreement on the value of CLI search without a clear scope of what that search would do. Just listing matches over the package names would be less useful than structured metadata search for example, but the latter makes a lot of assumptions about the availability of structured metadata!)
However, I get a lot of mileage out of package repository search with package managers like pacman, apt, brew, winget, chocolatey and npm.
> I think the biggest blocker on CLI search isn’t infrastructure
It's why it was shut down, the API was getting hammered and it cost too much to run at a reasonable speed and implement rate limiting or whatever.
Sort of: the original search API used a POST and was structured with XML-RPC. PyPI’s operators went to great efforts to scale it, but that wasn’t a great starting point. A search API designed around caching (like the one used on PyPI’s web UI) wouldn’t have those problems.
Search is an unbounded context and does not lend itself to caching very well, as every search can contain anything
And anyway, hit rates are going to be pretty good. You're not taking arbitrary queries, the domain is pretty narrow. Half the queries are going to be for requests, pytorch, numpy, httpx, and the other usual suspects.
2. apt repositories are cryptographically signed, centrally controlled, and legally accountable.
3. apt search is understood to be approximate, distro-scoped, and slow-moving. Results change slowly and rarely break scripts. PyPI search rankings change frequently by necessity
4. Turning PyPI search into an apt-like experience would require distributing a signed, periodically refreshed global metadata corpus to every client. At PyPI’s scale, that is nontrivial in bandwidth, storage, and governance terms
5. apt search works because the repository is curated, finite, and opinionated
You can Merklize “what files exist”; you can’t realistically Merklize “what should rank for this query today” without freezing semantics and turning CLI search into a hard API contract.
(Which isn’t to say I disagree with you about scale not being the main issue, just to offer some nuance. Another piece of nuance is the fact that distributions are the source of metadata but users think in terms of projects/releases.)
Why would you build a dedicated tool for this instead of just using a search engine? If I'm looking for a specific keyword in some project's very long README I'm searching kagi, not npm.
I'd expect that the most you should be indexing is the data in the project metadata (setup.py). That could be unbounded but I can't think of a compelling reason not to truncate it beyond a reasonable length.
(Note PyPI can’t index metadata from a `setup.py` however, since that would involve running arbitrary code. PyPI needs to be given structured metadata, and not all distributions provide that.)
Even including those, it's what? Sub-20-30GB.
The moment you expose that same service to a ubiquitous CLI like pip, the workload changes qualitatively.
PyPI has the /simple endpoint that the CDN can handle.
It’s PyPI philosophy that search happens on the website and pip has aligned to that. Pip doesn’t want to make a web scraper understandably so the function of searching remains disabled
For simple use cases, you have the web search, and you can curl it.