Hacker News Reader

My preferred fix for Yarn dependency tracking is to use zero installs[1] as there is no command to run and dependencies can be exactly what ships in the repo and nothing more (with the right flags). If accepting PRs from third-parties, the check-cache flag can run in CI to validate checksums from untrusted contributors — plus, you know, reading dependency source code when you have the time or reason to do so would be great within a PR review also.

I wish more tools were able to concisely show you the differences between dependencies, but… sometimes dependencies have binaries and at that point you might need to fork or clone and build your own version of a dependency. I’d suggest only using dependencies when you can read and understand the code, but there are always limits. I can’t think of the last time I thought of a glibc dependency except when using Alpine or compiling something for Cgo. But it’s still something to keep in mind: that sometimes your project will be simpler and easier with fewer, smaller dependencies where you can read the code in full.

1: https://yarnpkg.com/features/zero-installs#does-it-have-secu...

Great tips for application developers. Though this still exposes you to potential transitive dependency supply-chain attacks on first-time installs, though it's likely very low risk at this point, especially if you're careful as you've described.

For library authors that want to take reasonable precautions yarn makes it very hard. Though some suggest leaving transitive dependency updates up to the consumer makes yarn more secure, as they can update to the fixed packages sooner.. but I don't believe this benefit actually manifests often.

I guess the moral of the story is that semver is really great on paper, but people suck at it, sometimes on purpose.

In the extreme case, though, a library author can run their own build toolchain and create an executable npm module with zero declared dependencies, then version that as many times as they desire as long as their test cases guarantee that the new build still passes and dependencies have been thoroughly checked.

By publishing the source code to git, those who desire to use a specific version could alternately embed the git URL in their dependencies instead of the pre-built npm package.

Yarn Pack command can be used to run a build command[1] and Yarn Publish[2] in turn calls Yarn Pack, if I understand this correctly.

1: https://yarnpkg.com/advanced/lifecycle-scripts

2: https://yarnpkg.com/cli/npm/publish/#gatsby-focus-wrapper

That said… I agree that further work might be necessary here. I remember the first time I built an npm package and was confused when my lockfile in the package was completely ignored for dependency resolution. I don’t think this problem has gotten much attention, but the solution above might substantially solve the problem, if your dependencies are small enough to embed easily. It’s not ideal though, and I’d welcome other perspectives.

We do this for Ganache, except for 6 direct dependencies not authored by us, for some technical reasons. We do it twice as we ship a browser version and a node version. Our bundle size is not exactly ideal because of it. We'll be reducing our bundle size (the tarball downloaded from npm) soon and will likely increase it again shortly after by bundling these last 6 too.

Perhaps there’s an alternative to reduce the bundle size further: use code splitting or something like the shared module bundling that Webpack does for you when you have multiple entrypoints relying on the same shared code (if over a threshold minimum size) to reduce how much duplicated code is shipped multiple times for different entrypoints (e.g. uses or platforms or pages).

That said, the miracle of compression (and Yarn PnP not needing to uncompress) means you could have duplicate code and it won’t cost you much at all.

Another option would be to maintain your own dependencies by code splitting at the npm module level, but that could be considered an API-breaking change, I suppose, if the goal is to reduce how much is distributed by splitting out platforms.

You could also publish one layer of dependencies, probably, and still have exact versions pinned, but that would require maintaining CI build tools for your dependencies to ensure they are entirely built with no further dependencies or relying on republishing prebuilt no-dependency binaries in your own namespace.

I am reminded of how before tree-shaking became commonplace, it was routine to export libraries as lots of tiny npm packages, one per function sometimes, and import just the functions you needed as their own packages. That was taking it too far, and ECMAScript Modules (ESM) standards have somewhat replaced it, though for best tree-shaking you have to ensure your JS is actually modular and has no global state or side-effects when importing, which means you end up breaking some JS code when turning on Closure Compiler’s advanced mode, for example.

But we’re deep in the build optimization weeds now… the big advantage to pre-compiling libraries though is that you don’t have to tell others what build toolchain to use and instead provide standard ES5, ES6 or whatnot, already pre-compiled and ready for use (or… maybe, further tree-shaking…)

Do you regret using node/npm on such a high-risk installation?

When it was first built (6ish years ago? before I took over) there weren't any other practical options if you wanted an Ethereum simulator in JavaScript, which is a very common use case of ours.

Though now that wasm is popular and there are Rust implementations of Ethereum, I still think being able step into and debug from within a JavaScript application is valuable.

I think the ecosystem has just drank too much semver Kool-Aid.

So no, I don't regret it, and I don't think the problem is unique to npm, I think the problem is exacerbated by its popularity and addiction to loose semver (the default when installing a package by name).

Amen on semver. If you start pulling at that thread, the npm people start realising what a big mess they’ve gotten themselves in.

It's maddening the crypto world relies on Javascript so heavily. As a language it would never be my first, second, or even third choice for financial services.

I didn't know that about yarn transitives, do you have a link explaining more about the problem? I assumed a yarn.lock would.. lock, interesting to know more.

I assume the author is talking about the following problem:

I build library X and ship it with a lockfile to lock all transitive dependencies.

User installs library X, even though I ship a lock file, it isn’t respected. (I assume this makes Resolution way to complex)

If you ship a library where everything is locked, one should consider vendoring their dependencies. I think this is what npm does itself. (Or used to do). Or consider reducing your dependencies.

> I assume this makes Resolution way to complex

I can’t speak to Yarn’s motivation, but I suspect for most Yarn users it’s not dissimilar to my own: pinning exact versions of transitive dependencies can be a big source of “audit fatigue” with NPM. It’s fine that authors wish to limit the surface area of version ranges they support, but for downstream users this can often mean being essentially held hostage with potential exposure to vulnerabilities and no reasonable alternative.

Yarn should, like NPM, default to honoring all pinned dependencies. Both now have facilities to override those defaults (resolutions/overrides respectively), which is important. But there should also be a more gentle middle ground of “yes I want to assume this library meant to use semver, I understand I’m slightly relaxing the contract in the lockfile” in the form of a CLI flag… rather than manually writing potentially hundreds of lines of JSON.

> Or consider reducing your dependencies.

I mean, that sounds nice in theory. But the whole number of dependencies thing is a systemic design problem more than anything. No one is installing 1000s of dependencies. They’re installing ~5-10 and getting 1000s.

> Yarn should, like NPM, default to honoring all pinned dependencies. Both now have facilities to override those defaults (resolutions/overrides respectively), which is important. But there should also be a more gentle middle ground of “yes I want to assume this library meant to use semver, I understand I’m slightly relaxing the contract in the lockfile” in the form of a CLI flag… rather than manually writing potentially hundreds of lines of JSON.

Yes, this! I want this, and for all nose package managers to share a lock file format.

If a dependency includes it's own yarn.lock yarn will ignore it. It DOES respect the local lock file though, so it's not all bad.

ah ok so as a library author you can't actually enforce _your_ assumptions about transitives, bc user of the library local lock file overrides but as a library user with a local lock file that I craft and care about it is effective. That is a relief.

Yes. It's not a relief for everyone, especially library authors, as bugs can happen only on your particular set of transitive dependencies. What a fun time those days were.

Anyway, semver makes me sad sometimes. Haha

Wouldn't it be more practical to do development inside a container or VM? Locking deps won't be enough if you haven't also audited them.

Sure, but the valuable crypto credentials need to get used somewhere and the paranoid (justifiably-so) will note that VM escapes vulnerabilities, while not common, do exist, and aren't just hypothetical.

The current best practice, as I see it, is is a disposable chromebook, backed by a cold wallet which holds the keys to the multi-million dollar hoard.

I appreciate your thoughtfulness, and someone who thinks deeply about things with a touch of self-deprecation is the exact kind of conversation I enjoy at parties ;)

> Meanwhile, yarn, a popular npm alternative, will NOT respect a package author's wishes to lock transitive dependencies

Do you mean locking transitive dependencies across an upgrade of a direct dependencies? How does npm do that?

How did you accomplish this practically? using '=' in package.json + shrinkwrap or vendoring in dependencies?

We bundle most, and ship npm-shrinkwrap.json for these ones we don't. The shrinkwrap is pruned (via a a custom script, not npm's prune command, because of reasons ) at publish time, as npm installs all dev dependencies if they exist in the shrinkwrap file (despite being marked as "dev": true in said shrinkwrap).