Hacker News Reader

Comment by justinsaccount | Hacker News Reader

If anyone really cared, they would fix the broken ecosystem that encourages people to redownload the same packages millions of times.

Picked a popular package at random, webpack. npm says version 5.88.2 released 3 months ago has 5,992,398 downloads in the last 7 days.

I don't know how anyone can look at that see it as anything other than a massive failure.

Fast connections and free bandwidth have caused people to completely ignore the fact that every time some CI pipeline runs, npm goes off and downloads 100MB of dependencies. Dependencies that haven't changed since the pipeline last ran 30 seconds ago.

npm could fix this by aggressively rate limiting clients that have already downloaded the same package multiple times, but I guess as long as the vc funding is paying the bandwidth bill it's not a problem, and those "millions of downloads" make you look good.

View on HN

    > Fast connections and free bandwidth have caused people to completely ignore the fact that every time some CI pipeline runs, npm goes off and downloads 100MB of dependencies. Dependencies that haven't changed since the pipeline last ran 30 seconds ago.

Maybe it's just me, but I've always thought it was well known best practice to cache your deps[0].

I'm pretty certain that this can be achieved with most CI/CD tools.

https://docs.github.com/en/actions/using-workflows/caching-d...

I've seen a lot of pipelines that simply don't bother. Or maybe they tried, but the caching isn't working and since the build works in the end no one notices an extra 30 seconds.

Elaborate, please.

The vast majority of those are from CI on ephemeral cloud instances.

Do you think CI should not be run?

Or CI should be run, but not on ephemeral cloud instances?

Or CI should be run on ephemeral cloud instances, but the packages should be cached using a separate service from npmjs.com (e.g. S3)? If so, what makes this other service preferable?

> Or CI should be run on ephemeral cloud instances, but the packages should be cached using a separate service from npmjs.com (e.g. S3)? If so, what makes this other service preferable?

Yes, you should vendor external dependencies.

A build should ideally not require internet access to complete.

People learned nothing from leftpad.

> A build should ideally not require internet access to complete.

You've got a non-internet CI with non-internet source code repository with non-internet vendored dependencies??

Technically possible, but I call BS.

> You've got a non-internet CI with non-internet source code repository with non-internet vendored dependencies??

Vendored dependencies are pulled down from an internal s3 bucket (and cached locally) before the build starts, the rest of the build runs with no internet access.

Look how nix does this, it's basically the same.

Woosh.

Reply on news.ycombinator.com