Hacker News Reader

Comment by cedws | Hacker News Reader

Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.

View on HN

It's not even funny:

  $ ll /nix/store/*-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/*
  /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/linux:
  .r-xr-xr-x 129k root  1 Jan  1970 xsel

  /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/windows:
  .r-xr-xr-x 444k root  1 Jan  1970 clipboard_i686.exe
  .r-xr-xr-x 331k root  1 Jan  1970 clipboard_x86_64.exe

(clipboardy ships executables and none of them can be run on NixOS btw)

Are they reproducible? Shipping binaries in JS packages is dodgy AF - a Jia Tan attack waiting to happen.

The executables are vendored in the repo [0].

[0] https://github.com/sindresorhus/clipboardy/tree/main/fallbac...

I don't know why, but clipboard libraries tend to be really poorly implemented, especially in scripting languages.

I just checked out clipboardy and all they do is dispatch binaries from the path and hope it's the right one (or if it's even there at all). I think I had a similar experience with Python and Lua scripts. There's an unfunny amount of poorly-written one-off clipboard scripts out there just waiting to be exploited.

I'm only glad that the go-to clipboard library in Rust (arboard) seems solid.

That's on the package publishers, not NPM. They give you an `.npmignore` that's trivially filled out to ensure your package isn't full of garbage, so if someone doesn't bother using that: that's on them, not NPM.

(And it's also a little on the folks who install dependencies: if the cruft in a specific library bothers you, hit up the repo and file an issue (or even MR/PR) to get that .npmignore file filled out. I've helped folks reduce their packages by 50+MB in some cases, it's worth your own time as much as it is theirs)

It's much better to allowlist the files meant to be published using `files` in package.json because you never know what garbage the user has in their folder at the time of publish.

On a typical project with a build step, only a `dist` folder would published.

Not a fan of that one myself (it's far easier to tell what doesn't belong in a package vs. what does belong in a package) but that option does exist, so as a maintainer you really have no excuse, and as a user you have multiple MR/PRs that you can file to help them fix their cruft.

> On a typical project with a build step, only a `dist` folder would published.

Sort of, but always include your docs (readme, changelog, license, and whatever true docs dir you have, if you have one). No one should need a connection for those.

You might be interested in e18e if you would like to see that change: https://e18e.dev/

They’ve done a lot of great work already.

Does this replace ljharb stuff?

One of the things I like about node_modules is that it's not purely source code and it's not purely build artifacts.

You can read the code and you can usually read the actual README/docs/tests of the package instead of having to find it online. And you can usually edit library code for debugging purposes.

If node_modules is taking up a lot of space across a bunch of old projects, just write the `find` script that recursively deletes them all; You can always run `npm install` in the future when you need to work on that project again.

Yep, I wrote a script that starts at a root `node_modules` folder and iterates through to remove anything not required (dotfiles, Dockerfile, .md files, etc) - in one of our smaller apps this removes about 25Mb of fluff, some packages are up to 60-70mb of crap removed.

Totally agree with you. I wish npm did a better job of filtering the crap files out of packages.

At least, switch to pnpm minimize the bloat

I just installed a project with pnpm about 120 packages mostly react/webpack/eslint/redux related

with prod env: 700MB

without prod env: 900MB

sadly the bloat cannot be avoided that well :/

pnpm stores them in a central place and symlinks them. You’ll see the benefits when you have multiple projects with a lot of the same packages.

You'll also see the benefit when `rm -rf`ing a `node_modules` and re-installing, as pnpm still has a local copy that it can re-link after validating its integrity.

As someone who mostly works in Java it continues to floor me that this isn’t the default. Why does every project I work on need an identical copy of possibly hundreds of packages if they’re the same version?

I also like Yarn pnp’s model of leaving node_modules as zip files. CPUs are way faster than storage, they can decompress on the fly. Less disk space at rest, less disk slack, less filesystem bookkeeping.

Every single filesystem is way faster at dealing with one file than dozens/hundreds. Now multiply that by the the hundreds if does, it add up.

I believe I knocked 10% off of our node_modules directory by filing .npmignore PRs or bug reports to tools we used.

Now if rxjs weren’t a dumpster fire…

Reply on news.ycombinator.com