Hacker News Reader

Comment by BobbyTables2 | Hacker News Reader

How are the download rates of these things so high?

Are there 1000 people running 100 CI pipelines/day where downloads aren’t cached?

Yep, every single transitive downstream consumer pulls a fresh copy on every CI run. It's kind of ridiculous, and it's like this in every modern ecosystem.

For example, this is a completely empty rust crate, which I started years ago and never released, and it's still downloaded multiple times per day... https://crates.io/crates/ruble

Could it be caused by Crater runs?

https://github.com/rust-lang/crater

Oh so you're the reason we got btleplug!:P

https://github.com/deviceplug/btleplug

fwiw the downloads are probably automated crates.io indexers.

I'm confused. Why did you publish this? Who "uses" it?

I published it purely to placeholder the name (and then life got in the way, and I never released the actual library). Nobody uses it at all, but some CI system is still polling at regular intervals...

Wonder if someone has an internal crate named the same, but yours is getting pulled down for some tertiary use (license check, etc).

Stuff like Docker that defaults to downloading things from Internet repos is kinda scary. There isn’t always a clear dividing line between the thing as a “tool” and the thing as a “repo”.

At least with “git”, I know they I’m not going to end up cloning an artifact from an Internet repo just because I made a small typo…

But if no other crate depends on your crate and nobody directly uses it, why would any CI system download it?

I think some of the CI systems just monitor all published crates. For example, docs.io potentially pulls it to see if there is any new documentation

So someone could reserve all the pronounceable crate names?

Yeah, it’s a known problem in the (non-namespaced) crates.io ecosystem. If someone else wants this name for their Bluetooth LE crate, I’ll happily hand over the keys, but obviously not every crate-squatter is so principled

Yep. True webscale is astronomical levels of inefficient and wasteful usage of bandwidth and CPU.

It doesn't help that container images typically require you to "opt in" to sharing a cache with the host system. Docker even has a special file mount type for it. I find that people on average don't have a lot of interest in making such things work, they see that it's not available right in front of them and shrug and say "oh well" because ultimately it doesn't hurt them much.

Local caching is also a royal pain when one has multiple CI runners, besides the hassle of Docker related configuration.

Do they cache to shared storage and worry about concurrency? Do they only have local storage in various differing states?

Yeah I'd say most people view it as such, until they hit (docker hub) rate limits.

It seems to happen in the Python world, too. Check out the stats for the most commonly downloaded packages from PyPI (https://pypistats.org/top), and also consider which packages are on the list.

(Pip used to be right up there next to setuptools. It's still pretty popular — https://pypistats.org/packages/pip — but uv has clearly changed the equation.)

Have you noticed how much stuff breaks when Github or NPM is down? People do not cache, they do not mirror, modern development for large parts are just people YOLO-ing it.

Yes

Reply on news.ycombinator.com