My failed attempt to shrink all NPM packages by 5%
evanhahn.com
evanhahn.com
I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes).
It would have been interesting to see what eventually played out if the author hadn't closed the RFC themselves. It could have been the sort of thing that eventually happens after 2 years, but then quietly makes everybody's lives better.
They soft-rejected by requiring more validation than was reasonable. I see this all the time. "But did you consider <extremely unlikely issue>? Please go and run more tests."
It's pretty clear that the people making the decision didn't actually care about the bandwidth savings, otherwise they would have put the work in themselves to do this, e.g. by requiring Zopfli for popular packages. I doubt Microsoft cares if it takes an extra 2 minutes to publish Typescript.
Kind of a wild decision considering NPM uses 4.5 PB of traffic per week. 5% of that is 225 TB/week, which according to my brief checks costs around $10k/week!
I guess this is a "not my money" problem fundamentally.
That's the market price though, for Microsoft its a tiny fraction of that.
I do agree that 10k/week is non-negligible. Perhaps that means the people responsible for the 10k weren't in the room?
And I can buy 225 TB of bandwidth for less than $2k, I assume Microsoft can get better than some HN idiot buying Linode.
Even so, $2k a week is at least one competent FTE.
Checksum matters aside, wouldn't that turn the 5% bandwidth savings into an almost double bandwidth increase though? IMHO, considering the complexity to even make it a build time option, the author made the right call.
I found it reasonable.
The 5% improvement was balanced against the cons of increased cli complexity, lack of native JS zopfli implementation, and slower compression .. and 5% just wasn't worth it at the moment - and I agree.
>or at least I think they warrant further discussion
I think that was the final statement.
This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm.
You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to make any changes, until you experience your very own "oh man, I really thought this change was perfectly safe and now my entire customer base is trashed" moment, and then suddenly things like "hey, we need to consider how this affects old signatures and the speed of decompression and just generally whether this is worth the non-zero risks for what are in the end not really that substantial benefits".
I do not say this as the wise Zen guru sitting cross-legged and meditating from a position of being above it all; I say it looking at my own battle scars from the Perfectly Safe things I've pushed out to my customer base, only to discover some tiny little nit caused me trouble. Fortunately I haven't caused any true catastrophes, but that's as much luck as skill.
Attaining the proper balance between moving forward even though it incurs risk and just not changing things that are working is the hardest part of being a software maintainer, because both extremes are definitely bad. Everyone tends to start out in the former situation, but then when they are inevitably bitten it is important not to overcorrect into terrified fear of ever changing anything.
But it is possible to do it more gradually, I.e. by sneaking it in with a new API that's used by new npm version or similar.
But it was his choice to make, and it's fine that he didn't feel enough value in pursuing such a tiny file size change
5% is 5% at any scale.
I could never get Docker to work on my ADSL when it was 2 Mbps (FTTN got it up to 20) though it was fine in the Montreal office which had gigabit.
If that moves the needle on your ability to purchase the car, you probably shouldn't be buying it.
5% is 5%.
I'm sure you can use your imagination to substitute "lunch" and "car" with other examples where the absolute change makes a difference despite the percent change being the same.
Even taking it literally... The 5% might not tip the scale of whether or not I can purchase the car, but I'll spend a few hours of my time comparing prices at different dealers to save $2500. Most people would consider it dumb if you didn't shop around when making a large purchase.
On the other hand, I'm not going to spend a few hours of my time at lunch so that I can save an extra $1 on a meal.
- Doing 1 hour of effort to save 5% on your $20 lunch is foolhardy for most people. $1/hr is well below US minimum wage. - Doing 1 hour of effort to save 5% on your $50k car is wise. $2500/hr is well above what most people are making at work.
It's not about whether the $2500 affects my ability to buy the car. It's about whether the time it takes me to save that 5% ends up being worthwhile to me given the actual amount saved.
The question is really "given the person-hours it takes to apply the savings, and the real value of the savings, is the savings worth the person-hours spent?"
Also, this is exactly what you company is doing, paying you to "read the book" so they don't have to.
Here the scale of time is larger and does make the 5$ significant, while it isn't at the scale of a few days.
Given his numbers, let's say he saves 100Tb of bandwidth over a year. At AWS egress pricing... that's $5,000 total saved.
And arguably - NPM is getting at least some of that savings by adding CPU costs to publishers at package time.
Feels like... not enough to warrant a risky ecosystem change to me.
Even that's addressable though if there's motivation, since something like transcoding server side during publication just for popular packages would probably get 80% of the benefit with no client-side increase in publication time.
NPM uses at least 5 petabytes per week. 5% of that is 250 terabytes.
So $15,000 a week, or $780,000 a year in savings could’ve been gained.
For reference, total bandwidth used by all 5000 packages is 4_752_397 GiB.
Packages >= 1GiB bandwidth/week - That turns out to be 437 packages (there's a header row, so it's rows 2-438) which uses 4_205_510 GiB.
So 88% of the top 5000 bandwidth is consumed by downloading the top 8.7% (437) packages.
5% is about 210 TiB.
Limiting to the top 100 packages by bandwidth results in 3_217_584 GiB, which is 68% of total bandwidth used by 2% of the total packages.
5% is about 161 TiB.
Less than 1% of top 5000 packages took 53% of the bandwidth.
5% would be about 127 TiB (rounded up).
The more bandwidth that Cloudflare needs, the more leverage they have at the peering table. As GitHub's largest repo (the @types / DefinitelyTyped repo owned by Microsoft) gets larger, the more experience the owner of GitHub (also Microsoft) gets in hosting the world's largest git repos.
I would say this qualifies as one of those cases, as npmjs is hosted on Azure. The more resources that NPM needs, the more Microsoft can build towards parity with AWS's footprint.
Everyone benefits, the only cost is a ultra microscopic time on the front end, and a tiny cost on the client end, and for a very significant number of users, time and money saved. The examples of compression here...
It needs a novel JS port of a C compresison library, which will be wired into a heavily-used and public-facing toolchain, and is something that will ruin a significant number of peoples' days if it breaks.
For me, that kind of ask needs a compelling use case from the start.
Back in my Java days, most even small-time dev shops had a local Maven registry that would pass through and cache the big ones. A CI job, even if the "container" was nuked before each build, would create maybe a few kilobytes of Internet traffic, possibly none at all.
Now your average CI job spins up a fresh VM or container, pulls a Docker base image, apt installs a bunch of system dependencies, pip/npm/... installs a bunch of project dependencies, packages things up and pushes the image to the Docker registry. No Docker layer caching because it's fresh VM, no package manager caching because it's a fresh container, no object caching because...you get the idea....
Even if we accept that the benefits of the "clean slate every time" approach outweigh the gross inefficiency, why aren't we at least doing basic HTTP caching? I guess ingress is cheap and the egress on the other side is "someone else's money".
You can and should have your own caching proxy for all your builds but local caches are evil.
with prod env: 700MB
without prod env: 900MB
sadly the bloat cannot be avoided that well :/
I also like Yarn pnp’s model of leaving node_modules as zip files. CPUs are way faster than storage, they can decompress on the fly. Less disk space at rest, less disk slack, less filesystem bookkeeping.
Every single filesystem is way faster at dealing with one file than dozens/hundreds. Now multiply that by the the hundreds if does, it add up.
They’ve done a lot of great work already.
Now if rxjs weren’t a dumpster fire…
(And it's also a little on the folks who install dependencies: if the cruft in a specific library bothers you, hit up the repo and file an issue (or even MR/PR) to get that .npmignore file filled out. I've helped folks reduce their packages by 50+MB in some cases, it's worth your own time as much as it is theirs)
On a typical project with a build step, only a `dist` folder would published.
> On a typical project with a build step, only a `dist` folder would published.
Sort of, but always include your docs (readme, changelog, license, and whatever true docs dir you have, if you have one). No one should need a connection for those.
$ ll /nix/store/*-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/*
/nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/linux:
.r-xr-xr-x 129k root 1 Jan 1970 xsel
/nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/windows:
.r-xr-xr-x 444k root 1 Jan 1970 clipboard_i686.exe
.r-xr-xr-x 331k root 1 Jan 1970 clipboard_x86_64.exe
[0] https://github.com/sindresorhus/clipboardy/tree/main/fallbac...
I just checked out clipboardy and all they do is dispatch binaries from the path and hope it's the right one (or if it's even there at all). I think I had a similar experience with Python and Lua scripts. There's an unfunny amount of poorly-written one-off clipboard scripts out there just waiting to be exploited.
I'm only glad that the go-to clipboard library in Rust (arboard) seems solid.
You can read the code and you can usually read the actual README/docs/tests of the package instead of having to find it online. And you can usually edit library code for debugging purposes.
If node_modules is taking up a lot of space across a bunch of old projects, just write the `find` script that recursively deletes them all; You can always run `npm install` in the future when you need to work on that project again.
Its not always obvious who has the most important use cases. In the case of NPM they are prioritizing the user experience of module authors. I totally see how this change would be great for module consumers, yet create potentially massive inconvenience for module authors.
Interesting write-up
Also, since this is a case of having something compressed once and decompressed potentially thousands of times, it seems like the perfect tool for the job.
This is changing every build in every CI system to make it slower.
The fix would then have to be some variation of:
a) Stop testing (so often)
b) Stop bundling before testing
c) Publish to a different package manager
- all of which would affect the overall quality and quantity npm modules.
But yes- you could bundle less, but that would be a disadvantage, particularly if a bundle suddenly fails and you don’t know which change caused it. But maybe that’s not a big deal for your use case.
Starting with smaller data means everything ends up smaller. It's the same decompression algorithm in all cases, so it's not some special / unoptimized branch of code. It's yielding the same data in the end, so writes equal out plus or minus disk queue fullness and power cycles. It's _maybe_ better for RAM and CPU because more data fits in cache, so less memory is used and the compute is idle less often.
It's relatively easy to test decompression efficiency if you think CPU time is a good proxy for energy usage: go find something like React and test the decomp time of gzip -9 vs zopfli. Or even better, find something similar but much bigger so you can see the delta and it's not lost in rounding errors.
This means an optimization like this is virtually guaranteed to be a net positive on the receiving end, since you always save a bit of time/energy when downloading a smaller compressed file.
Zopfli is easiest to think of as something that just tries harder than gzip to find matches and better encodings. Much harder.
decompression speed is linear either way.
It's easiest to think of decompression as a linear time vm executor[1], where the bytecoded instructions are basically
go back <distance> bytes, output the next <length> bytes you see, then output character <c>
(outputting literal data is the instruction <0,0,{character to output}>)
Assuming you did not output a file larger than the original uncompressed file (why would you bother?), you will, worst case, process N bytes during decompression, where N is the size of the original input file.
The practical decompression speed is driven by cache behavior, but it thrashes the cache no matter what.
In practice, reduction of size vs gzip occurs by either finding larger runs, or encodings that are smaller than the existing ones.
After all, if you want the compressed file to shrink, you need output less instructions somehow, or make more of the instructions identical (so they can be represented in less bits by later huffman coding).
In practice, this has almost exclusively positive effects on decompression speed - either the vm has less things to process (which is faster), or more of the things it does look the same (which has better cache behavior).
[1] this is one way archive formats will sometimes choose to deal with multiple compression method support - encode them all to the same kind of bytecode (usually some form of copy + literal instruction set), and then decoding is the same for all of them. ~all compression algorithms output some bytecode like the above on their own already, so it's not a lot of work. This doesn't help you support other archive formats, but if you want to have a bunch of per-file compression options that you pick from based on what works best, this enables you to still only have to have one decoder.
https://quixdb.github.io/squash-benchmark/
There even more obscure options:
The package lock checksum is probably a more solvable issue with some coordination.
The benefit of doing this though is less immediate - it will take a few years to show payoff and these kinds of payoffs are not typically made by the kind of committee decisions process described (for better or worse).
If accepted, it might have been a good stepping stone too. A chance to get to know everyone and their concerns and how they think.
So if you wanted to see how this works (proposal + in prod) and then come back later proposing something bigger by switching off zip that would make sense to me as a possible follow up.
Back then if you wanted to use a particular neural network it was meant for a certain version of Tensorflow which expected you to have a certain version of the CUDA libs.
If you had to work with multiple models the "normal" way to do things was use the developer unfriendly [1][2] installers from NVIDIA to install a single version of the libs at a time.
Turned out you could have many versions of CUDA installed as long as you kept them in different directories and set the library path accordingly, it made sense to pack them up for conda and install them together with everything else.
But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time.
If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression.
[1] click... click... click...
[2] like they're really going to do something useful with my email address
For Paper, I'm planning to cache both the wheel archives (so that they're available without recompressing on demand) and unpacked versions (installing into new environments will generally use hard links to the unpacked cache, where possible).
> If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression.
FWIW, in my testing LZMA is a big win (and I'm sure zstd would be as well, but LZMA has standard library support already). But there are serious roadblocks to adopting a change like that in the Python ecosystem. This sort of idea puts them several layers deep in meta-discussion - see for example https://discuss.python.org/t/pep-777-how-to-re-invent-the-wh... . In general, progress on Python packaging gets stuck in a double-bind: try to change too little and you won't get any buy-in that it's worthwhile, but try to change too much and everyone will freak out about backwards compatibility.
git clone https://github.com/google/zopfli.git
cc -O2 zopfli/src/zopfli/*.c -lm
Comparing to gzip isn't really worth it. Combine pigz (threaded) with zlib-ng (simd) and you get decent performance. pigz is used in `docker push`.
For example, gzipping llvm.tar (624MB) takes less than a second for me:
$ time /home/harmen/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-13.2.0/pigz-2.8-5ptdjrmudifhjvhb757ym2bzvgtcsoqc/bin/pigz -k hello.tar
real 0m0.779s
user 0m11.126s
sys 0m0.460s
It is almost 2700x slower than the state of the art for just 6.8% bytes saved.
That is impressively slow.
In my opinion even the 28x decrease in performance mentioned would be a no-go. Sure the package saves a few bytes but I don't need my entire pc to grind to a halt every time I publish a package.
Besides, storage is cheap but CPU power draw is not. Imagine the additional CO2 that would have to be produced if this RFC was merged.
> 2 gigabytes of bandwidth per year across all installations
This must be a really rough estimate and I am curious how it was calculated. In any case 2 gigabytes over a year is absolutely nothing. Just my home network can produce a terabyte a day.
Because the authors mentioned package, Helmet[1], is 103KB uncompressed and has had 132 versions in 13 years. Meaning downloading every Helmet version uncompressed would result in 132*103KB = 13.7MB.
I feel like I must be missing something really obvious.
Edit: Oh it's 2GB/year across all installations.
[1]: https://www.npmjs.com/package/helmet?activeTab=versions
Another personal win is that you got a very thorough understanding of the people involved and how the outreach parts of the RFC process works. I've also had a few fail, but I've also had a few pass! Always easier to do the next time
https://github.com/fhanau/Efficient-Compression-Tool
These days if you’re going to iterate on a solution you’d better make it multithreaded. We have laptops where sequential code uses 8% of the available cpu.
Repetition eliminating compression tends to be inherently sequential. You'd probably need to change the file format to support chunks (or multiple streams) to do so.
Because of LZ back references, you can't LZ compress different chunks separately on different cores and have only one compression stream.
Statistics acquisition (histograms) and entropy coding could be parallel I guess.
(Not a compression guru, so take above with a pinch of salt.)
But zopfli appears to do a lot of backtracking to find the best permutations for matching runs that have several different solutions. There’s a couple of ways you could run those in parallel. Some with a lot of coordination overhead, others with a lot of redundant calculation.
https://jamiemagee.co.uk/blog/honey-i-shrunk-the-npm-package...
and the related HN discussion from that time:
> When it was finally my turn, I stammered.
> Watching it back, I cringe a bit. I was wordy, unclear, and unconvincing.
> You can watch my mumbling in the recording
I watched this, and the author was articulate and presented well. The author is too harsh!
Good job for trying to push the boundaries.
The conversation started and ended at the word cache.
The fact NPM keeps things in node_modules unzipped seems wild to me. Filesystems are not great at hundreds of thousands of little files. Some are bad, others are terrible.
Zip files are easier to store, take up less space, and CPUs are faster than disks so the decompression in memory is probably faster reading the unzipped files.
That was one of my favorite features of Yarn when I tried it - pnp mode. But since it’s not what NPM does it requires a shim that doesn’t work with all ps mage’s. Or at least didn’t a few years ago.
TS releases a non-preview version every few months, so using 2.5 minutes for compression would work.
Such effort would be better spent preparing npm/node for a future where packages with a lower-bound npm version constraint can be compressed with zstd or similar.
Every feature you add to a program is complexity, you need to really decide if you want it.
Could verify package integrity by hashing contents rather than archives, plus digital signatures for recompressed versions. Only kicks in for frequently downloaded packages once compression is ready.
Benefits: No npm changes needed, opt-in only, potential for big bandwidth savings on popular packages. Main tradeoff is additional verification steps, but they could be optional given a digital signature approach.
Curious if others see major security holes in this approach?
For security reasons, it's usually better to hash the compressed file, since it reduces the attack surface: the decompressor is not exposed to unverified data. There have already been vulnerabilities in decompressor implementations which can be exploited through malformed compressed data (and this includes IIRC at least one vulnerability in zlib, which is the standard decompressor for .gz).
BTW npm decompressed all packages anyhow because it lets you view the contents these days on its website.
Working approach, but in the end I abandoned the project - I doubt people care about such js size savings.
I'm sure it works great when you've got high-speed internet, but might break things unacceptably for users on mobile or satellite connections.
This part is obviously trivially solvable. I think the same basic idea is going to at some point make it but it’ll have to be through explicit annotations first and then there will be tooling to automatically do this for your code based upon historical visits where you get to tune the % of visitors that get additional fetches. Also, you could probably fetch the split off script in the background anyway as a prefetch + download everything rather than just 1 function at a time (or even downloading related groups of functions together)
The idea has lots of merit and you just have to execute it right.
Note that most of the unused code is located in libraries, not in the app code directly.
This would cause the bundler to inject a split point & know how to hide that + know what needs bundling and what doesn’t. GWT pioneered almost 20 years ago although not a fan of the syntax they invented to keep everything running within stock Java syntax: https://www.gwtproject.org/doc/latest/DevGuideCodeSplitting....
The tooling producing such annotaion based on historical visits whould constantly change the annotated set of functions, I suspect, whith app versions evolving.
Note, the inactive code parts are very often in 3rd party libraries.
Some are also in your own libraries shared between your different application. So the same shared library whould need one set of split-annotated functions for one app, and another set for another app. So these conflicting sets of annotations can not live in the library source code simultaneously.
As soon as the user executes a rarely used function, the information is about this fact is sent to the server and it includes this function into the active set to be send to future users.
In the video I manually initiate re-generation of the "active" and the "rest" scripts, but the most primitive MVP was supposed to schedule re-generation of the scripts when receiving info that some previously unseen functions are executed in browser.
Obviously, if the idea is developed further, the first user's experience may also be improved - a spinner shown. Pre-loading the inactive set of functions in background may also be considered (pros: it allows to avoid latency for users ho invoke rare functionality, cons: we lose the savings of the traffic and browser memory and cpu for compiling the likely unneded code).
(BTW, further development of the idea includes to splitting the code more granularity than just "active" / "inactive". E.g. active in the first 5 seconds after opening the page loaded immediately, likely to be active soon, active but rarely called - the later two these parts definitely need to be loaded in background)
50% is just a big O of the original size :)
If we consider overall Internet traffic, which is dominated by video and images, the size of javascript transmitted is negligible.
Savings of memory and cpu for end user's browser? Maybe, but our software at all layers is so bloated, without need, just carelessly, that I'm not sure working of such javascript savings is useful - any resources saved will be eaten by something else immediately.
For an application developer or operator, the 50% savings of javascript size are probably not worth dealing with some tool that dynamically prunes the app code, raising questions about privacy and security. I though through the security and privacy questions, but who would want to even spend attention on these considerations?
As I mention in the README at github, a Microsoft researcher investigated the same approach earlier, but Microsoft haven't taken it anywhere.
There was a commercial company offering this approach of javascript minification as a product, complete product. As well as other optimizations, like image size, etc. Their proxy embedded special "agent" code into the app which inspected the device and reported to server what image sizes are optimal for user, what js functions are invoked. And the server prepared optimized versions of app for various devices. Javascript was "streamed" in batches of only functions needed by the app.
Now the company is dissolved - that's why the website is unavailable. Wikipedia says they were bought out by Akamai - https://en.wikipedia.org/wiki/Instart. But I don't see any traces of this approach in the today's Akamai offerings.
I contacted several companies, in CDN business and others, trying to interest them in the idea and get very modest funding for a couple more months of my time to work on this (I was working on this in the end of a long break from payed work and was running out of savings). Didn't find anyone ready to take part.
This all may be signs that possibility of such an optimization is not valuable enough for users.
Then the info about the called functions is sent back to the server.
(Only the functions never seen to be called are instrumented, the known active functions are not instrumented).
I think people forget that, when you’re copying the same images to dozens and dozens of boxes, any improvement starts to add up to real numbers.
Reduced is the size of javascript loaded into the end user's browser.
I'm surprised it hasn't been raised when talking about saving 2Tb/year only for React. It represents costs, which doesn't seem to be an issue, but also computing power and storage. (Event with a higher/longer computing power due to slower compression, it's done once per version, which isn't really comprable to the amount of downloads anyway)
Hard to calculate the exact saving, but it would represent a smaller CO2 footprint..
That's a strong argument that 'if it isn't broke, don't fix it."
LOts of numbers being thrown around, you add up tiny things enough times you can get a big number. But is npm package download the thing that's tanking the internet? No? Then this is a second- or thirt-order optimization.
Is it possible to modify "gzip -9" or zlib to invoke zopfli? This way everyone who wants to compress better will get the extra compression automatically, in addition to npm.
There will be an increase in compression time, but since "gzip -9" is not the default, people preferring compression speed might not be affected.
It's been almost 30 years since bzip2 was released and even now not everything can handle tar.bz2
Earlier in my career, I managed to use Zopfli once to compress gigabytes of PNG assets into a fast in-memory database supporting a 50K+ RPS web page. We wanted to keep it simple and avoid the complexity of horizontal scaling, and it was OK to drop some rarely used images. So the more images we could pack into a single server, the more coverage we had. In that sense Zopfli was beneficial.
> Zopfli is written in C, which presents challenges. Unless it was added to Node core, the CLI would need to (1) rewrite Zopfli in JS, possibly impacting performance (2) rely on a native module, impacting reliability (3) rely on a WebAssembly module. All of these options add complexity.
Wow! Who's going to tell them that V8 is written in C++? :)
This is a canard. zopfli is written in portable C and is far more portable than the nodejs runtime. On any hardware/OS combo that one can build the nodejs runtime, they certainly can also build and run zopfli.
Also for bigger companies this is not really a "big" problem as they usually have in-house proxies (as you cannot rely on a 3rd party repository in CI/CD for multiple reasons (security, audit, speed, etc)).
The tar contains a whole bunch of related files that probably compress a lot better together, and it’s only one file on disk for the file system to keep track of. It’s gonna be a lot more efficient.
When compiling you’re probably gonna touch all the source files anyway right? So if you load the zip into memory once and decompress it there it’s probably faster than loading each individual file and decompressing each of them, or just loading each individual file raw.
The future of compression is likely shared dictionaries, basically recognize the file type and then use a dictionary that is optimized for that file type. E.g. JavaScript/TypeScript or HTML or Rust or C++, etc. That can offset the problems with small files to a large degree.
But again, this is a complex problem and it should be dealt with orthogonally to npm in my opinion.
I also checked and it should be possible for npm to enable file system compression at the OS level on both Windows and MacOS.
Also if it was dealt with orthogonally to npm, then it could be used by pip, and other package systems.
- I don't see the problem with adding a C dependency to a node project, native modules are one of the nicest things about node.js
- Longer publishing time for huge packages (eg. typescript) is a bigger problem but I don't think it should impact a default. Just give the option to use gzip for the few outliers.
So it is basically downloading a few hundred kb dictionary every year ?
But past that the most common words in a React project are going to be very different from a Vue project right?
> For example, I tried recompressing the latest version of the typescript package. GNU tar was able to completely compress the archive in about 1.2 seconds on my machine. Zopfli, with just 1 iteration, took 2.5 minutes.
[1] https://github.com/npm/rfcs/pull/595#issuecomment-1200480148
My question of course would be, what about LZ4, or Zstd or Brotli? Or is backwards compatibility strictly necessary? I understand that GZIP is still a good compressor, so those others may not produce meaningful gains. But, as the author suggests, even small gains can produce huge results in bandwidth reduction.
5% sounds like a good deal
Why is this substantial? My understanding is that packages shouldn't be touched once published. It seems likely for any change to not apply retroactively.
Maybe it would make sense to only apply this improvement in images that are a) either very big or b) get downloaded at least million times each year or so. That would cover most of the savings while leaving most packages and developers out of it.
If you want an example of where these things go badly:
the standard compression level for rpms on redhat distros is zstd level 19.
This has all the downsides of other algorithms - it's super slow - often 75-100x slower than the default level of zstd[1]. It achieves a few percent more compression for that speed. compared to even level 10, it's like 0-1% higher compression, but 20x slower.
This is bad enough - the kicker is that at this level, it's slower than xz for ~all cases, and xz is 10% smaller. The only reason to use zstd is because you want fairly good compression, but fast.
So here, they've chosen to use it in a way that compresses really slowly, but gives you none of the benefit of compressing really slowly.
Now, unlike the npm case, there was no good reason to choose level 19 - there were no backwards compatibility constraints driving it, etc. I went through the PR history on this change, it was not particularly illuminating (it does not seem lots of thought was given to the level choice).
I mention all this because it has a real effect on the experience of building rpms - this is why it takes eons to make kernel debuginfo rpms on fedora. Or any large RPM. Almost all time is spent compressing it with zstd at level 19. On my computer this takes many minutes. If you switch it to use even xz, it will do it about 15-20x faster (single threaded. if you thread both of them, xz will win by even more, because of how slow the setting is for zstd. If you use reasonable settings for zstd, obviously, it achieves gigabytes/second in parallel mode)
Using zopfli would be like choosing level 19 zstd for npm. While backwards compatibility is certainly painful to deal with here, zopfli is not likely better than doing nothing. You will make certain cases just insanely slow. You will save someone's bandwidth, but in exchange you will burn insane amounts of developer CPU.
zopfli is worse than level 19, it can often be 100x-200x slower than gzip -9.
Doesn't npm support insanely relaxed/etc scripting hooks anyway?
If so, if backwards compatibility is your main constraint, you would be "better off" double compressing (IE embed xz or whatever + a bootstrap decompressor and using the hooks to decompress it on old versions of npm). Or just shipping .tar.gz's than, when run, fetch the .xz and decompress it on older npm.
Or you know, fish in the right pond - you would almost certainly achieve much higher reductions by enforcing cleaner shipping packages (IE not including random garbage, etc) than by compressing the garbage more.
[1] on my computer, single threaded level 19 does 7meg/second, the default does 500meg/second. Level 10 does about 130meg/second.
> The only reason to use zstd is because you want fairly good compression, but fast
I would think having fast decompression is desirable, too, especially for rpms on redhat distros, which get decompressed a lot more often than they get compressed, and where the CPUs doing decompression may be a lot slower than the CPUs doing the compression.
And zstd beats xz in decompression times.
https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_c...
Just two tables for comparison, first one shows only the decompression for Firefox RPM, second shows compression time, compressed size, and decompression for a large RPM.
You'd think there'd be more data.
In this case, yes, zstd has faster decompression , but xz decompression speed is quite fast, even before you start using threads. I have no idea why their data found it so slow.
Here's an example of this: https://web.archive.org/web/20231218003530/https://catchchal...
Even on large compressed archives, xz decompression times does not go into minutes - even on a mid range 16 year old intel CPU like being used in this test. Assuming the redhat data on this single rpm is correct, i would bet it's more related to some weird buffering or chunking issue in the rpm's compressor library usage than actual xz decompression times. But nobody seems to have bothered look at why the times seemed ridiculous, at all, they just sort of accepted them as is.
They also based this particular change on the idea that they would get similar compression ratio to xz - they don't, as i showed.
Anyway, my point really wasn't "use xz", but that choosing zstd level 19 is probably the wrong choice no matter what.
There own table, which gives you data on 1 whole rpm, shows that zstd level 15 gave them compression comparable to xz (on that RPM, it's wrong in general), at compression speed similar to xz (also wrong in general, it's much slower than that).
It also showed that level 19 was 3x slower than that for no benefit.
Result: Let's use level 19.
Further the claim "Users that build their packages will experience slightly longer build times." is total nonsense - their own table shows this. If you had an RPM that was 1.6gb, but took 5 minutes to build (not uncommon, even for that size, since it's usually assets of some sort), you are now taking 30 minutes, and spending 24 of it on compression.
Before it took ... 3 minutes to do compression.
Calling this "slightly longer build times" is hilarious at best.
I'll make it concrete: their claim is based on building Firefox and compressing the result, and amusingly, even there it's still wrong. Firefox RPM build times on my machine are about 10-15 minutes. Before it took 3 minutes to compress the RPM. Now it takes 24.
This is not "slightly longer build times". Before it took 30% of the build time to compress the RPM.
Now it takes 24 minutes, or 2.5x the entire build time.
That is many things, but it is not a "slightly longer build time".
I'll just twist the knife a little more:
RPM supports using threading for the compressors, which is quite nice. It even supports basing it on the number of cpus you have set to use for builds. They give examples of how to do it, including for level 19:
/usr/lib/rpm/macros:
# "w19T8.zstdio" zstd level 19 using 8 threads
# "w7T0.zstdio" zstd level 7 using %{getncpus} threads
Despite this - they did not turn on threads in the result...
/usr/lib/rpm/redhat/macros:%_binary_payload w19.zstdio
All this to say - i support NPM in being careful about this sort of change, because i've seen what happens when people aren't.
Could be because they want reproducible builds.
Second, zstd is fully deterministic in multithreaded cases. It does not matter what threading you select, it will output byte for byte identical results.
See a direct answer to this question here: https://github.com/facebook/zstd/issues/2079
I believe all of their compressors are similarly deterministic regardless of number of threads, but i admit i have not checked every one of them under all conditions.
If they had questions, they could have, you know, asked, and would have gotten the same answer.
But that just goes back to what i said - it does not appear this change was particularly well thought out.
But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do.
For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And I would have quit at the first roadblock.
What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line.
Do you want to get a change in, or not?
Is this a project working with the community or not?
> Just release your work
What motivation exists to optimize package formats if nobody uses that package format? There are no benefits unless it’s in mainline.
> Nobody in open source wants to be told what to do
He’s not telling anybody to do work. He is sharing an optimization with clear tradeoffs - not a new architecture.
> You cannot force your improvement into the whole group
Nope, but communication is key. “put up a PR and we will let you know whether it’s something we want to pursue”.
Instead they put him through several levels of gatekeepers where each one gave him incorrect feedback.
“Why do we want to optimize bandwidth” is a question they should have the answer to.
If this PR showed up on my project I would say “I’m worried about X,Y,Z” we will set up a test for X and Y and get back to you. Could you look into Z?
Says who?
Says an engineer? Says a product person?
NPM is a company with 14 employees; with a system integrated into countless extremely niche and weird integrations they cannot control. Many of those integrations might make a professional engineer's hair catch fire - "it should never be done this way!" - but the real world is that the wrong way is the majority of the time. There's no guarantee that many of the downloads come from the official client, just as one example.
The last thing they need, or I want, or any of their customers want, or their 14 employees need, is something that might break backwards compatibility in an extremely niche case, anger a major customer, cause countless support tickets, all for a tiny optimization nobody cares about.
This is something I've learned here about HN that, for own mental health, I now dismiss: Engineers are obsessed with 2% optimizations here, 5% optimizations there; unchecked, it will literally turn into an OCD outlet, all for things nobody in the non-tech world even notices, let alone asks about. Just let it go.
Hell. Even 15 employees larping as a corporation is going to be inefficient.
what you and NPM are telling us, is that they are happy to take free labor, but this is not an open source project.
> Engineers are obsessed with 2% optimizations here
Actually in large products these are incredible finds. But ok. They should have the leadership to know which bandwidth tradeoffs they are committed to and tell him immediately it’s not what they want, rather than sending him to various gatekeepers.
> Actually in large products these are incredible finds.
In large products, incredible finds may be true; but breaking compatibility with just 0.1% of your customers is also an incredible disaster.
Yes. But in this story nothing like that happened.
Just because it's safe in a lab provides no guarantee it's safe in production.
The conclusion of this line of reasoning is to never make any change.
If contributions are not welcome, don’t pretend they are and waste my time.
> can no longer decompress fast enough
Already discussed this in another thread. It’s not an issue.
"If it ain't broke, don't fix it."
I absolutely disagree with you. If the world took more of those 5% optimisations here and there everything would be faster. I think more people should look at those 5% optimisations. In many cases they unlock knowledge that results in a 20% speed up later down the line. An example from my past - I was tasked with reducing the running speed of a one shot tool we were using at $JOB. It was taking about 15 minutes to run. I shaved off seconds here and there with some fine grained optimisations, and tens of seconds with some modernisation of some core libraries. Nothing earth shattering but improvements none the less. One day, I noticed a pattern was repeating and I was fixing an issue for the third time in a different place (searching a gigantic array of stuff for a specific entry). I took a step back and realised that if I replaced the mega list with a hash table it might fix every instance of this issue in our app. It was a massive change, touching pretty much every file. And all of a sudden our 15 minute runtime was under 30 seconds.
People used this tool every day, it was developed by a team of engineers wildly smarter than me. But it had grown and nobody really understood the impact of the growth. When it started that array was 30, 50 entries. On our project it was 300,000 and growing every day.
Not paying attention to these things causes decay and rot. Not every change should be taken, but more people should care.
> Says an engineer?
I prevent cross-site scripting, I monitor for DDoS attacks, emergency database rollbacks, and faulty transaction handlings. The Internet heard of it? Transfers half a petabyte of data every minute. Do you have any idea how that happens? All those YouPorn ones and zeroes streaming directly to your shitty, little smart phone day after day? Every dipshit who shits his pants if he can't get the new dubstep Skrillex remix in under 12 seconds? It's not magic, it's talent and sweat. People like me, ensuring your packets get delivered, un-sniffed. So what do I do? I make sure that one bad config on one key component doesn't bankrupt the entire fucking company. That's what the fuck I do.
If/when NPM has a problem - storage costs are too high, or transfer costs are too high, or user feedback indicates that users are unhappy with transfer sizes - then they will be ready to listen to this kind of proposal.
I think their response was completely rational, especially given a potentially huge impact on compute costs and/or publication latency.
That said, I think NPM’s response was totally correct - explain the problem and the tradeoffs. And OP decided the tradeoffs weren’t worth it, which is totally fair.
...which makes it all the weirder that security isn't any better, as in, publishing a package can be done without a review step on the npm side, for example. I find it strange that they haven't doubled down on enterprise offerings, e.g. creating hosted versions (corporate proxies), reviewed / validated / LTS / certified versions of packages, etc.