Ride down into JavaScript dependency hell | Hacker News Reader

Ride down into JavaScript dependency hell

blog.appsignal.com

133 pointsnikolalsvk143 comments

How to write a silly post about JavaScript dependencies:

- Don't mention other languages, lest you reveal that most of them have similar dependency bloat problems.

- Talk about devDependencies and dependencies without considering how they might be entirely different, say… one kind gets bundled, the other doesn't.

- Always use npm, so you can highlight duplicates. Don't use yarn.

- Adopt a narrow definition of 'dependency hell' so you can focus on the thing that JavaScript does imperfectly (minimizing number of dependencies) and avoid talking about the things that it does well (handling conflicting deps, solving diamond deps, resolving dependency chains)

- Don't worry about explaining how any of the issues you're discussing have real-world implications. Have companies been failing because they run out of disk space because of node_modules?

- Take automated security audits extremely seriously. A backtracking bug in the argument parser for the web framework's local CLI tool? Of course it's not exploitable, when… it's a web app, and the CLI tool is something you run locally. But add it to the count as if it is!

How to downplay the severity of the problem:

Ignore the ridiculously small functionality scope that acceptably qualifies for a js package in a way that does not exist in any other ecosystem and ads a big multiplier in front of all the other risks involved.

If these claims were true, we would have seen some examples by now. Instead, we get some tired invocations of "leftpad!" along with some more tired FUD. Who has been hacked because they used javascript instead of some other "more mature" language?

There is more to dependency risk than 'getting hacked' - correctness, maintainability, IP due diligence all play a role.

Yes. I've had several apps that can basically never be upgraded due to complete insanity deep in their dependency trees.

I'm pretty cautious about third party dependancies, but some of those libraries' libraries are not, and it's really hard to see that coming.

If you expect to live with code for 5+ years (and I understand that's not everyone, but it's almost always me), this is a big problem.

We've all lived with code that was 5+ years old, even if not all those years were ours. b^) Often that meant wrangling envars to point to the right ancient binaries, assuming we could somehow get those binaries installed. None of that is required with javascript. If a dependency is causing problems, just replace it with a different one. (npmjs.com has lots of packages!) Or, write something yourself. In five years you'll find enough spare time.

> - Don't mention other languages, lest you reveal that most of them have similar dependency bloat problems.

The post contains a graph with the package counts for other languages [1]

1 https://d33wubrfki0l68.cloudfront.net/7481900a584f733e7e9db7...

Is node's biggest crime is that it's easy to publish packages? Is this an argument for intentionally complicating package systems?

The issue with NPM is that packages tend to be more granular than other languages.

For example: The npm package "is-even" depends on the "is-odd" package. (In case it's not clear, this is a real package, not a made-up example.) In most other languages, these would be bundled together into an Integer Utilties package, if they're not already part of the standard library. In NPM, they are separate packages.

I believe there is, or was, a technical reason for NPM to have very fine-grained imports. Perhaps something about not supporting partial imports? I can't exactly find it. But the point is: NPM having more packages does not translate to a larger and more vibrant ecosystem. In some cases it simply means accomplishing the same thing requires a larger dependency graph.

That's not to say it's inherently a bad thing. But that's at least why it's not inherently a good thing either.

[1] https://www.npmjs.com/package/is-even

> In most other languages, these would be bundled together into an Integer Utilties package, if they're not already part of the standard library.

This is the root of all the many issues. There is no standard library. So things that are trivial to write but also dumb to rewrite every time you need them (left-pad a great example) turn into tiny dependencies.

> For example: The npm package "is-even" depends on the "is-odd" package. (In case it's not clear, this is a real package, not a made-up example.) In most other languages, these would be bundled together into an Integer Utilties package, if they're not already part of the standard library. In NPM, they are separate packages.

People should not be using these functions at all, in any language, in a standard library or otherwise. To convert the even/oddness of an integer to a boolean in Javascript you can use !!(n % 2), which returns true for odd inputs, and false for even inputs. If you want to be extra explicit you can temporarily store the result of that expression in a meaningfully named variable. If it will just be used in a conditional, then the !! can be skipped.

Depending on a third-party package for this kind of thing is insanity. But the blame lies with programmers who use this package, not with the programming language.

If the standard library includes a variant for these or any other functions, it's probably idiomatic to use them. They handle edge cases, are better optimized and they're explicit.

There is no way that any Javascript library function is "more optimized" than just using `n % 2`, which is also plenty descriptive.

If this is very performance sensitive you can try using the somewhat more obscure (because many people have minimal experience with bitwise operators) `n & 1` instead.

Given an integer input, thare are no "edge cases" to worry about here. If you expect floating point inputs, you need to think carefully about what the correct output should be anyway. A library isn’t going to save you.

(If your "edge case" is that someone might want to know whether a string is even or odd... well, you have other problems.)

It's funny, because in the case we're talking about, there appears to be edge case behavior if the number is a "safeInteger." While I agree in principle I would never import something as granular as "is-odd," just researching this specific package revealed an edge case someone ran into and solved, and you don't have to worry about as a dev if you take it off the shelf. https://github.com/jonschlinkert/is-odd/blob/master/index.js...

Your "isodd" function should probably not be doing bounds checking and throwing an error when it encounters large even integers.

If you have a use case where you need to know if you multiplied two large integers and got a result larger than 2^53, then you should do it explicitly before you get to the point of checking if your number is even or odd.

These comparisons are so bad.

NPM is hands-down the most popular packaging ecosystem online. So of course it has more packages than other ecosystems, more developers use it. It's not a good comparison.

I would not be surprised to find that JS bloat tends to be more severe than other languages, because Javascript makes it a lot easier to import and publish packages. I would also not be surprised if Node package management ended up being a bit worse, again because Node lowers the barrier of entry for inexperienced programmers to publish packages on impulse, even if they don't plan to maintain them. And its just undeniable that Javascript is a heavy intro language, so the average developer quality (especially around responsible dependency management) is probably less than other languages.

But the package count stats need to just die. When we talk about dependency bloat, useful stats would be:

- What is the average/median number of (deduplicated) dependencies in a Node program vs a Rust/Python/Ruby project?

- What is the average/median number of lines of code in a Node package, minus its dependencies? How does that compare to the average/median package in Rust/Python/Ruby?

- What is the average/median number of outdated dependencies in a Node package, vs the average/median in Rust/Python/Ruby?

- What are the average/median number of different authors across a Node package's dependencies vs Rust/Python/Ruby?

- What percentage of the Node ecosystem actually gets used? Are we looking at a long tail of mostly ignored packages, or is usage fairly spread out and diverse?

Heck, even this mostly useless graph would be better if it just adjusted for the number of users for each platform. It's pretty tough get that data, but there are sources you could look at to try and guess, including StackOverflow's dev surveys[0].

The "how many packages are there" metric means nothing when it's quoted in isolation from other data. It's like claiming Windows developers are vastly more productive than Mac developers because more Windows software exists.

[0]: https://insights.stackoverflow.com/survey/2019#overview

Also keep in mind it's far easier to publish on npm and that's a good thing.

It literally takes days to setup shit to get published on maven central and even then you can't set and forget. It's a pain in the ass.

Yesterday I had to use the web fullscreen API. Didn't work on an iPad, so I installed npm package screenfull that handles browser incompatibilities. Works like a charm.

These kind of things are specific for JavaScript because of the mentioned incompatibilities, and of course increase nr packages used. But I'm very happy they exist.

They also strategically avoid mentioning the payoff which is that your project is entirely self-contained. `node_modules` is at once your complete development kit, build tool, compiler, and collection of runtime dependencies.

Once you check out a node project with a package.json you can reasonably assume you can install it and run it anywhere. A docker image for a node project should just be "install node && npm build". You're side-stepping the entire dev ops nightmare. Find me another language with JavaScript's popularity and that level of simplicity. I'd go as far to argue that this is what happens to any massively popular language that has a functional package manager from day one.

I'm totally baffled by this statement.

Find me a mainstream language that's more complicated than this? "Checkout and build" works pretty much everywhere? I mean, you need the right version of mvn or pip or whatnot... just like you need the right version of npm.

If anything, node/npm deserves special mention for NOT having reliable repeatable builds. Part of this is that a lot of packages rely on native code. But most of it is that the ecosystem culture defaults to "always grab the latest versions of everything".

And npm only got package-lock.json a few years ago, not "from day one". Prior to package-lock.json, builds were wildly unpredictable - like, expect breaking changes week to week even if you don't change anything at all in your code.

If you want an "entirely self contained" payoff, languages that produce static binaries are pretty hard to beat. Node is not that.

Heck, in javaland the paractice now is to put the build tool in the repo, as you will only update maven once in every decade git won't complain too much about the binary, or you can just update the version in the wrapper and put the binary on the git ignore. First time you run it, it will update the build tool.

Nowadays on javaland you should only need to have JDK and git/svn/hg/(wtv version control you use). And with the JDK, you can always download the most recent that java is backwards compatible.

Yes and now lock files exist?

Are you actually arguing over the state of JS development from 4 years ago?

I have pointed out that, contrary to the parent, Node is not special, and certainly wasn't "from day one" as claimed.

To put it simply to build a modern Java project you need the jdk + bazel + docker + maven + runtime classpath for a sub-set of npm's features (as far as I know you still can't include two versions of the same maven dependency).

For a Node project you just need node. npm is part of node, and yarn is optional. The repeatable builds was a real issue but that's been solved years ago.

That's not true for Java. Most require Maven and then it's mvn install (you need the JDK for maven, but that feels similar to saying you need node for npm, they effectively come together).

Npm's dev dependencies are analogous to Maven's plugins.

Bazel is an alternative to maven. Docker fulfills the same role as in the Node ecosystem. The runtime class path is an implementation detail that you need to concern yourself with if you wish to run your Java code without Maven (analogous to running your JavaScript code without npm).

The two forms of the same dependency is a limitation that stems from Java the language rather than its ecosystem.

FWIW, nearly every language ecosystem I know of is at the point of "download build tool, run build tool."

Worse than that: lock files aren’t lock files the way that they are in every other language package manager that has lock files. In Cargo, Bundler, and Mix, you specify a pessimistic version (~> 2.1) and you may get 2.1.1 or 2.3.0. But that version is _always_ the same for every developer because the lock file locks the version and you explicitly upgrade after that.

I recently had a case where a developer joined us on a project and he got a different version of a package than I did because the lockfile didn’t constraint the dependencies and sub-dependencies and everything else. (For that you have to pass an explicit parameter like `--ci` or `--frozen-lockfile` depending on which of three different package managers you use.)

Bollocks, I say.

> A docker image for a node project should just be "install node && npm build"

That defeats the purpose of having separate dev and runtime dependencies and results in a huge docker image filled with dev-time dependencies.

Also, other package managers do this fairly well. Java has maven and uber-jars (for self contained artifacts), python has pip and venv, etc.

Unless any library you use depends on a native module, then you are in for surprises.

> - Don't mention other languages, lest you reveal that most of them have similar dependency bloat problems.

no such problems in C, i promise you

Part of the issue is that JavaScript packages are often far far smaller than packages in other ecosystems.

"small packages are extremely common in the JavaScript npm package system. For example, in npm, 47% of the packages have 0 or 1 functions, and the average npm package has 112 physical lines of code. In contrast, the average Python module in the PyPI repository has 2,232 physical lines of code."

Source: "Vulnerabilities in the Core: Preliminary Report and Census II of Open Source Software", The Linux Foundation & The Laboratory for Innovation Science at Harvard, by Frank Nagle, Jessica Wilkerson, James Dana, Jennifer L. Hoffman. https://www.coreinfrastructure.org/programs/census-program-i...

(This is a contrast of citations from other works; you can see the specific citations in the report, but I'm trying to do this from a phone. In any case, the issue is the massive difference between JavaScript and everything else, and I think this quotation shows it nicely.)

iOS dependencies in Cocoapods are including less than 1 other packages on average (1). The most I see are Alamofire (a networking package people keep using for reasons that I do not fully understand) and Starscream (a Websocket implementation, that's pretty hard to write from scratch to be honest). Having unnecessary dependencies is pretty much frowned upon.

One of the reasons is that the iOS SDK is really complete by itself while Javascript doesn't even ship with a usable Date implementation. If somebody could just come up with a standard library for JavaScript that is really widely adopted I think the amount of packages can be cut back dramatically.

(1) totally based on gut feeling, not based on hard data

Java didn't have a decent date implementation until Java 8.

> If somebody could just come up with a standard library for JavaScript that is really widely adopted I think the amount of packages can be cut back dramatically.

As this article points out, I think that lodash is that for many people.

From what I read about it it’s more like syntactic sugar, not something alike the .Net core libraries.

Get your JS Core libraries once from a repo, reuse them in every application that uses the same ones

Java has had exact datetime implementations since what, 1.3? 1.2? Sometime in the ‘90s anyway. They weren’t fun to use but they worked fine, had the concept of multiple calendars etc...

JS’s version of Date in comparison has always been a toy.

JS's version of Date is pretty much exactly the same as java.util.Date: just a thin veneer around a long representing ms since the epoch. Yes, Java has had Calendar and DateFormat for years, but the API is generally recognized as being pretty horrible (how many bugs were caused by Date and SimpleDateFormat not being thread-safe?)

So the Java community coalesced around Joda time, not unlike how the JS community coalesced around moment.js. The big difference is that yes, Java eventually rolled these lessons learned into the java.time API, but now you're effectively stuck with 2 standards, Joda and java.time. Not sure if that's much of an improvement.

Google Apps Script! Heh...

What makes the js Date implementation unusable in your view? IMO it works pretty okay for most needs.

This is what happens with a wafer-thin standard library.

Yes and no.

Because of the small stdlib, you need dependancies, sure.

But why this idea that each dependency must be very small, forcing you to install a hundreds childs, which, themselves, will install a hundred until the pyramid of files start to turn node_modules into a stress test?

The problem is not small packages or huge dependency trees. It's the fact widely used packages have to be installed at all. This is compounded by the package manager which allows packages to have independent copies of their dependencies.

We should have a de facto standard library that gets distributed along with the language's reference implementation. Widely-used packages that have a stable and well-designed interfaces should be distributed by default. If something isn't being used anymore, it can be dropped from the standard library while still being available from the repositories.

The problem is not small packages. The problem is there are many of these small libraries trying to solve the same problem (extending the small standard library more). And when there are no clear winners your dependency tree will be full of packages trying to do the same.

One idea that probably got encoded in the culture is that this code would also have to ship to the client. By having small deps, nobody had to wait around for more advanced tree-shaking (and its idiosyncrasies) to enter the picture.

Make sense.

Which is weird. I would have expected have the leftpad scandal that the community would have found a balance.

To me, culture is a feature, and despite using JS a lot since it has a monopoly on the browser, the community culture never clicked with me.

I often feel like they are reinventing the wheel, not learning from other techs solving the same problems, under and over-engineering, regularly guilty of the XKCD 927 syndrom.

Few projects taste integrated, working together, clean. The stacks looked like hacked together. Agreement on best practices and conventions are timidly showing up. It also seems popular tools are just starting to stabilize and the breaking modifications rhythm to slow down. At least more than anywhere else I worked.

There is no way I can be objective saying this, but anyone with the same feeling?

There are several explanations:

1) JavaScript is a first language for many, many new developers, especially the ones not coming from a formal computer science program. This is because it's multi-paradigm, extremely hireable, runs on everything, runs on the web specifically, and other reasons. If you're only going to learn one language, it's a very good and versatile choice. So its community has a lot of inexperienced members.

2) Because it's such a focal point of the industry, it gets a lot of attention from major players as well and has been evolving rapidly in terms of core features. These features usually get tested out first in the form of packages.

3) Being a dynamic language it's very easy to extend and hack. The prototype system even allows you to modify the behavior of its core primitives.

4) The more experienced devs who use it are often dissatisfied with the qualities of the base language, and so take advantage of its flexibility to try and extend it into what they think a good language is (see TypeScript, Ramda, RxJS, etc).

The response to the leftpad scandal was for NPM to disallow packages being unpublished once they've been out for 72 hours, which really should've always been the case.

Why are small packages bad? Independent functions should be versioned and distributed independently. Otherwise we get several utility packages which are nothing but collections of independent functions.

Widely used packages should form the basis of a standard library that is distributed with the language itself.

> Why are small packages bad?

Small, separately maintained packages are bad in a an ecosystem where a final system can incorporate only one version of any given package because it increases the number of opportunities for version conflicts.

Now, it's true that there are some other concerns which weigh in favor of packages being at the minimum useful size, but those necessarily also weigh in favor of a package management system which allows each package to isolate its upstream dependencies without constraining it's downstream users. And not just allows, but facilitates it so that it is the norm, so that dependency conflicts aren't a thing.

I was under the impression that in the npm ecosystem, packages are reused if the version dependencies are compatible, and split if they are not.

Every separate package adds overhead and another maintainer that you've got to put your trust in. I'd rather have few packages of well trusted maintainers instead of a thousand packages from god knows who.

Personally, I've made it a habit to not add anything that requires trivial one liner garbage packages, which amounts to not installing any dependency that uses anything from Jon Schlinkert (so no Webpack). Unfortunately I'm stuck with gulp right now but with the next major rewrite, I'll also get rid of gulp and its 300 dependencies.

> I'd rather have few packages of well trusted maintainers instead of a thousand packages from god knows who.

Then the problem is the fact anybody can submit a package, not small packages.

Linux distributions solve this problem by having dedicated maintainers. Users of a distribution trust its maintainers when they use it. Software developers want the complete opposite: language-specific packages, instant and unrestricted package publication, containers, etc. Nobody wants to have to talk to a maintainer in order to get their software included in a distribution. Of course the result ends up being a mess.

What if Node's maintainers decided to distribute a curated set of packages alongside Node itself? The size of each individual package wouldn't really matter.

> What if Node's maintainers decided to distribute a curated set of packages alongside Node itself?

First, front-end has nothing to do with Node. Secondly, let’s say there’s some sort of front-end foundation who decides to provide a curated set of packages. A natural candidate would be create-react-app, which pulls over a thousand dependencies. That’s just one package you want to include. How the hell do you even start to curate such a beast?

> First, front-end has nothing to do with Node.

It was just an example.

> A natural candidate would be create-react-app, which pulls over a thousand dependencies. That’s just one package you want to include. How the hell do you even start to curate such a beast?

I don't know. How did Linux distributions do it? Arch Linux has over 10 thousand packages. Debian has over 50 thousand packages. Maybe people should ask the maintainers.

Point to me one package in Debian that depends on thousands more.

Oh and I was a maintainer for a major package manager back in the day for crying out loud (was responsible for overseeing the general health of the project as well as directly responsible for maybe a couple dozen individual packages). Never seen this kind of madness.

I don't know about Debian but Arch Linux has huge package groups and metapackages.

https://www.archlinux.org/groups/x86_64/kde-applications/

https://www.archlinux.org/groups/x86_64/pro-audio/

The Arch Wiki recommends the use of these huge packages. I don't see any reason why it wouldn't scale to a thousand packages or more.

I'm talking about one actual package that largely runs in a single process (or at least a single process group) with thousands of deps, not some sort of loosely related by category or by using the same framework (hell, we're talking about the framework itself here) package group where failure of one no name package doesn't affect anything else.

Groups are very different from dependencies. And a big meta-package is there for user convenience to get a bunch of stuff. It's almost never going to be depended on; if something needs a package or two from the group it will depend on that directly.

Have you considered using Rollup? I've personally found it delightful.

Already using rollup as a replacement for Webpack, but it seems to lag behind from time to time. Couldn't use certain new javascript features right away since it would throw an error.

Small packages means more packages, and more packages compounds the dependency graph which:

- makes dependency resolution harder

- means more points of failure

- makes auditing difficult

- makes install time longer

- leads to harder maintenance and upgrade

This also causes heterogeneity in the mass of your dependencies:

- it splits the resources for documentation, testing, tutorial, etc

- it ensures very weak integration between various building blocks, forcing everyone to rebuild glue code again and again

- it makes discovering the proper solution to your problem harder as you must chose a combination of dependencies instead of one

- it makes contributing to the project harder, especially to junior profiles, and increases the price of on-boarding

- eventually, it leads to a culture that pushes the "libs versus frameworks" so far you never get a decent framework for anything. This is why there is no JS equivalent to RoR or Django.

There is, of course, a balance to reach. You don't want a lib that does everything.

The package manager also has evolved to handle lots of dependencies.

In the article 19000 packages are installed in 40 seconds... My Django project with 100 times fewer dependencies takes longer in CI

Npm comes with an audit tool

I have never seen dependency resolution fail (since packages can have private copies), unlike pypi or rubygems.

So some of the downsides to a large dependency tree are mitigated. I'll add one more downside:

- more chances for shenanigans like left-pad to cause issues.

Don't get me wrong, I think 19000 dependencies is fucking nuts. NINETEEN THOUSAND.

> 19000 dependencies

Doesn't this happen because of package manager duplication? I think npm lets packages have their own copies of their dependencies. Since we have a lots of small, widely used packages, they get duplicated at numerous points in the dependency graph. The number of files and installation size explodes.

A big part is the development toolchain, which would be installed at a system level for most languages.

I started a vue project last night, npm install --production installs 4 packages. With dev dependencies, I get 2300 packages. Eslint, babel, webpack, etc bring in lots of luggage

BTW, I think 19000 is wrong, on a fresh node_modules I get:

    $ npm install gatsby
    ...
    + gatsby@2.20.18
    added 1773 packages from 733 contributors 
     and audited 23706 packages in 52.317s
    $ du -sh node_modules
    245M node_modules

Not sure where the "audited" number comes from, but its not the number of install packages. I get 2737 directories containing a package.json, 1477 of which are uniq.

`debug` appears 32 times!

I think that we are in a situation where the number of dependency doesn't have any meaning anymore for people working on web frontend projects. Recently while revamping parts of the CI/CD at $WORK, I found out that running npm install for one of our frontend project downloads around 1 million dependencies. And npm audit reports a completely ridiculous number of security issues with them.

It's just so absurd and nonsensical, nobody can understand what that even mean.

Seems to me to be a crazy huge liability, but quite a lot of people seem to be fine with that.

I don't know what to do from that information, but trying to create some awareness around the issue feels like talking about the number of operations per second a CPU does with someone who has zero idea what a realistic range is. It's seen as a high number, but doesn't have any meaning attached to it.

Edit: I don't have access to the project anymore to check more details (such as installation time), but I checked some personal notes I took when I found this out, the redacted output from npm audit is the following

    $ npm audit
    <REDACTED> vulnerabilities found - Packages audited: 927016
    Severity: <REDACTED> Low | <REDACTED> Moderate
    Done in 41.62s.

So slightly below the 1 million mark.

Edit 2: Now I'm wondering ... maybe the "packages audited" from npm audit doesn't mean "you have that number of packages, and I audited them". But if that's the case, that's a terrible UX, and I have zero idea what that number mean, which kind of support my point.

A million dependencies sound horribly excessive, even by JS standards.

That was my reaction too.

how long does that take to install? is there a local npm mirror / cache?

I mean npm only has slightly more than 1 million packages, last I checked.

This doesn't make even the slightest bit of sense. Did somebody somehow just import... everything by accident? Did npm hit a bug? Is there some infinite recursion going on that eventually hits a wall?

This obviously isn't representative.

1 million packages most of which have more than a single version.

How is this even possible?

How are there 1 million interesting packages?!

In my job we use like, I don't know, 20 libraries total, out of, say, 50 alternatives that could do the job. Including dependencies of the dependencies I don't think we reach anywhere near 100. Every dependency is discussed with the entire team when we add it.

How do you even have time to manage 1 million?!

Presumably almost all of the dependencies are transient dependencies, not direct ones.

I still can't fathom getting to a million though.

To quote the article:

"Luckily, most of them are using the same version of lodash, which just needs one lodash to install inside node_modules. This is often not the case with real-world production projects. Sometimes, different packages require different versions of other packages."

Npm lets you install several versions of the same package. This has the advantage of saving the dev the testing of their lib with a wide range of dependencies versions: it almost always works. Unfortunately, that means most devs just choose one version and call it a day. They build libs like one would build a project: as you were alone in the world.

I have a project with a total of eight dependencies. I walked away for six months as it was stable. I come back to add a few features and npm tells me I have over 38 000 vulnerabilities of different severity levels. So many that 'npm audit' just freezes up.

So there's that too.

My stuff breaks whenever I try upgrade it. People refactoring stuff.

These types of issues are the biggest reason why I avoid javascript/npm projects. I came back to a project after a few months and it was broken, had to rewrite some parts and upgrade other parts just to get it to run again.

I'm sorry but for all the flaws of the NPM ecosystem, version locking is not one (both `npm` and `yarn` do much much better than anything in Python). There's no reason a correctly set-up project would magically break with time.

I don't love NPM anymore than the next guy, but my blame will go to the dev in this case.

Python dependency management is a disaster. But I find myself using far fewer dependencies with Python projects so I suffer the pain less.

This is more of that "wet streets cause rain" logic we hear about...

No, version locking is a disaster in npm and yarn both (and ppm appears to do the same thing). For it to be _safe_, you have to opt into `--ci` or `--frozen-lockfile`.

Cargo, Mix, and Ruby’s Bundler _all_ do an infinitely better job because they don’t let dependencies upgrade on you behind the scene. Their lockfiles are really lockfiles. No ifs, ands, or buts.

`--frozen-lockfile` and the equivalent should be the DEFAULT behaviour, not this pseudo-locked nonsense that currently exists.

I cannot understand how people work with this. I work with many different technologies and try to avoid JS, but some times I have to. The past weeks I have worked on a React Native project that was written by someone else; what a horror show. I mean it wasn't the worst code by the previous guy but even in a few months a lot is simply broken and not 'best practice' anymore. Compared to most other environments I work with, iteration is somewhat faster when you finally have everything working, but the dependencies and ecosystem is horrible imho.

A bigger issue with all of it is that in my line of work, for the backend of firmware I cannot just depend on 3rd party libs as they will get audited, so I need to audit them before. Ofcourse most people don't have that issue.

I don't agree with you on this. The JS (or NPM) way makes it a pain to audit a project due to thousands of dependencies, but it's far easier to "have everything working".

`npm i && npm start` is as good as it gets in terms of "getting started" friction (with version locking): python is worse, ruby is worse, PHP is worse, Java is worse.

React-Native has more moving pieces indeed, but it's not really JS's (or NPM's) fault: you get all problems of iOS and Android at once. I will definitely agree with you that working on React-Native is a pain, to be able to work with modern tools need to know JS, optionally TS, Obj-C, Swift, Java, Kotlin, Ruby (for Fastlane) and Gradle's custom language. Insane.

I've done 3 years of Golang as my main language, my experience is:

- go build ./... builds everything from the current project

- go test ./... tests everything from the current project

- go install ./... installs everything from the current project

That's one of the best thing ever. I can go to any project, I know how to build, test, install, and can directly be productive.

On the other hand, I started learning C++ one year ago, and it's the exact opposite, each project has its own homemade build system that isn't supported by one platform or the other.

Golang does a lot better than other languages (but probably the language that allowed me to use the most unmaintained/oldest projects is Clojure, I can't remember/find out when the oldest breaking change to the language was), but the situation in Golang was piss poor not too long ago, when Golang libraries were all fetched from GitHub/Git sources without any way of specifying specific versions, you basically lived on the edge all the time. There was third party tools to work around this (and the ever present vendoring technique) but seems weird they didn't realize this until gomod started being a thought.

If only version locking weren’t fundamentally broken in JS package managers. I’ve said it a couple of other times on this, but `--frozen-lockfile` should be the default behaviour. Under no circumstances should any developer get a different package version than what is locked in the lockfile. This is the exact opposite of what npm, yarn, and pnpm do with their lockfiles (unless you use `--frozen-lockfile` or `--ci`).

All our JVM projects use gradle. We run `gradle build` and it resolves all dependencies and runs all tests. I don't understand the claim that node has the least getting started friction.

It's not really hell though, is it? Hell is infinite pain and agony. This is more like an occasional bother.

Yes, we get comically large numbers when we look through node_modules. But, real "dependency hell" is when you have situations that take unbounded manual effort to resolve.

How often do we end up with impossible circular dependencies? Or conflicting versions clobbering each other? Or non-deterministic builds introducing random behavior?

That all is commonplace even today with other platforms like python, and rarely an issue with node. I'd much rather occasionally sneer at the size of node_modules than any of that actual hell.

In my experience is the people who complain most audibly about npm are the same folks who think nothing of spending a few days debugging classpath issues.

Yes, exactly. I used to think isolating dependencies like npm/yarn do, where each package has its own copy of its own dependency tree (ignoring deduping), was crazy, and potential conflicts should be a forcing function to encourage you to minimize dependencies in the first place. But then I started using these systems and it's by far the least time I've ever spent worrying about deps. There are downsides to this and how it encourages dependency proliferation, and I still believe in minimizing your dependencies, but it's the least bad system I've seen.

What’s worse is that the whole installing multiple versions of the same package for different sub dependencies doesn’t make any sense when when the language supports singletons and module level variables with has no way of specifying which version to import.

I was shocked this behavior exists when a site broke by upgrading a sub dependency. Turned out they both installed react so they used a different “creatContext” and now there were two context instances instead of one.

React would normally be a peerDependency and so you shouldn't run into this problem if dependencies are specified properly.

If not, NPM module flattening should take care of it.

Then worst case, you can also tell webpack to bundle a specific installation.

There are some ways to solve these problems...

Sure (upvote) and we dealt with it accordingly after some confused digging. On a more general level I’ve never seen something like that in a package manager. It seems as though some of the language features (like the singleton I mentioned) would make it obvious that arbitrarily duplicating packages and scoping them to sub packages is a major no-no, as in most languages I’ve dealt with.

It seems like a weird thing about React, or maybe how you're using it, that this is a problem. Most packages, and the language itself, handle multiple simultaneous versions of dependencies without issue. If packages are doing the old-fashioned "stick a property on window" thing instead of just getting required or imported browserify-style, they can step on each others' toes (and it's totally possible that React is dumb enough to step on its own toes). It has been years since that was necessary. browserify or more recent tools like parcel or webpack should be used instead.

That’s not really relevant, right? No mater the loading/transpiration step, you lose a lot of language guarantees when you can load multiple of the same packages, including:

- function/object identity checking I.e “package.func !== package.func” in many cases places.

- module level singletons and registry (e.g createContext useContext)

- single initialization

- others I can’t think of

That’s not typical and I was surprised by it.

Loading multiple versions of the same package is just something that you can do in javascript. As complained about many other times ITT, any big node tool does this as a matter of course. Over on the browser, react isn't my favorite, but I'll admit that all that complexity serves some purpose. Maybe it would be nice if it were just a bit more complex so that global contexts could be backwards-compatible, but I guess it's not. Still, if you hire developers, they should be able to handle this sort of thing.

My concern with npm packages isn't the number and small size of the packages itself, but that most of them are maintained and owned by individuals who do work with little review by others and who could disappear any time. If an individual goes away or takes harmful action it takes time till this is noticed and till a surviving fork emerges.

If more libraries were maintained by groups in a shared/collaborative way, more of those risks would go away.

Sadly the alternative isn't much better: the old school model of not having packages often leads to libraries never getting updated at all. In C++ often adding a library is such a chore once you get through all the linking issues, as you battle the 4000 conflicting compiler switches, and compiler versions, that once it works you never want to change it again.

I think maybe the best solution we have so far are robust "batteries included" standard libraries like python, where even if they're huge theyre at least curated and maintained by a central group.

I feel like Bazel and Cmake have solved this sufficiently well. I regularly compile larger projects from scratch and usually the process is to call vcvars (I'm on Windows) and then Bazel and then an hour later I have a folder full of cached lib and dll files so that linking my actual executable only takes a second.

I would be so incredibly grateful to have something comparable in JavaScript that quickly checks static typing for my entire project. But the closest I have found is Rust/Elm, so not using JavaScript anymore.

That page itself is an example what's wrong with using JavaScript on the web: It's used in places where it's not necessary. In this case the site doesn't apply a stylesheet with JavaScript disabled.

JavaScript is great when it's used for enabling interactive websites, but nowadays more and more websites use it to break features, which don't need JavaScript at all.

express (common web framework) seems to have only 51 deps in lockfile

jekyll has 13

I don't disagree that lots of deps = supply chain risk

But there have always been variations between projects & kinds of projects re how many deps they pull in. Try following someone's ipython data science tutorial, it's requirements.txt for days

I think lack of a standard lib early on plus hipster functional culture made the '10s JS leaders like small monofunctional libs.

Focus on transpilation plus lack of tree-shaking in early versions of webpack may have also created an incentive for small things.

A python equivalent to "express" would be Flask https://palletsprojects.com/p/flask/

I see 4 dependencies in the setup.py, and they are all from the same team of maintainers (i.e the Pallet team):

- Werkzeug

- Jinja2

- itsdangerous

- click

https://github.com/pallets/flask/blob/master/setup.py

https://github.com/pallets/flask/network/dependencies

'pip freeze' results from a few other py web frameworks:

sanic = 18, fastapi = 3 but one of them is a 7MB download (pydantic)

yeah it's less, but also python has an amazing stdlib

To be fair, express is a production server itself whereas werkzeug is not, so you need to add something like gunicorn or uwsgi (yeah I know nginx supports uwsgi_pass natively). click is not necessary in production though (it's only used in the dev CLI).

That said, 4 dependencies is the norm in Python land, whereas 13 dependencies is an outlier in JS land.

Edit: oops, misread, gp said express has 51 deps, not 13.

The Python standard library is a lot more capable than the JS equivalents (Node or browser API). That eliminates the need for a lot of dependencies.

There's been a TC39 proposal for a JS stdlib for about 2 years now: https://github.com/tc39/proposal-javascript-standard-library

Express is a micro web framework, it is very small and not really usable on it's own (and it's not meant to be).

So what does NPM do differently than plugin managers from other languages? I can't name one where the dependency hell is this big.

My personal take having “grown up” with all of this - JS is one of the worst ecosystems for relying on external packages for everything. It’s more the culture than the tooling.

One of the things I love about Go is generally people are a little more forgiving to copying something around a couple times instead of making a lib for everything.

Inviting the question of why the culture hasn't sprouted a "package rationalizer" to produce a curated, tested, noise-filtered distribution of useful functionality.

Come to think of it, something that does for packages what jQuery did to bring (some) sanity to browser incompatibilities.

Because we can't even agree on whether React or Vue or Angular is best.

In fact this is the root of the issue. It's important to realize that `node_modules` for any project in Popular Framework X will be almost identical. Most of the packages are related to compiling, bundling, and the development environment and a small minority are actual client-side dependencies. Even the frameworks themselves are pretty light -- React has a grand total of 2 dependencies. It's packaging the entire toolchain that causes most of the bloat.

Go in particular has an amazing standard library and very clear guidelines on how to write code. JavaScript does not have either of those things so it promotes people releasing competing libraries with similar functionality written in various different styles.

Copying code is pinning a dependency, committing to never update it, and hiding the evidence.

Perhaps.

That's not always true.

Many a time I've forked a dependency for one reason or another, and kept it up to date with upstream. Sure, it's a few more steps to update, but allows for deep customizations without waiting for original authors to support it (if ever).

Another reason to fork/copy code, is that its development has ceased long ago. In that case, it totally makes sense to carry on the work, in-house at first.

Regarding the article, sometimes copying code keeps the entire codebase easier to understand and manage without having an external dependency.

I guess that's not impossible, but it seems to reinvent git submodules without the tool support I'd want, and I think explicitly depending on our fork of their library would be clearer and safer.

> explicitly depending on our fork of their library would be clearer and safer.

I complete agree. To temper my comment, I've seen plenty of bad examples of copying dependencies, bundling old un-updated versions, with undocumented changes. In such case, it's true that copying leads to a worse kind of dependency.

I’d say compared to other languages there’s been two big issues:

1. A not insignificant number of packages are either polyfilling things in browsers, or providing a consistent (or ‘isomorphic') API for certain things across both browsers and Node.js, or adding things that should have been in a standard library.

2. A lot of packages still seem to be distributed primarily as CommonJS and not ES Modules. CommonJS makes tree-shaking harder than it should, so it was often just easier to break what should have been a single library into many smaller pieces.

Hopefully in the not too distant future library maintainers get around to reading the Node.js docs[0] fix some of this.

[0] https://nodejs.org/api/esm.html

It's the users. And money.

JavaScript is super popular and used by companies so there are lots of people releasing their own open source project in an attempt to hustle into a well-paid job by looking more experienced than they really are.

I'd say it's the same reason why there are so many bad Java and Php tutorials.

C++ for comparison is shielded from this because it's not the kind of programming language that you would learn in a weekend seminar to improve your salary.

Other languages have for most part a good standard library. In C# for instance you can make a small/medium project with only a few third party libraries because the base class libraries (BCL) contains lot of data structures and classes used for common use case (encryption, unicode, file operations, etc.)

If you want the 3D view of Gatsby dependencies without waiting for the initial 2D view to populate, you can go here:

https://npm.anvaka.com/#/view/3d/gatsby

200 megabytes is not that bad. I’ve seen it rise above 700 MB easily.

A reasonable complaint! Instead of npm, try pnpm [0]. It uses file links to avoid duplication.

[0] https://pnpm.js.org/

This wouldn't be much of a problem if developers weren't as reliant on frameworks as JS developers are — who for some reason love to blame others for "reinventing the wheel". What do people even use lodash for? Do you really need a whole library because you can't write a function to sort an array? This all reminds me of the npm left-pad scandal. To me the problem was not that a single package broke thousands of projects, but that thousands of developers depended on a package for such a simple feature.

If you think this is just npm, try running "cargo update" on a older Rust project.

I can't help but find it ironic that in middle of this oft-repeated JS bitch fest the author's own company is promoting their NPM package.

You have to market differently to different people. TFA is intended for devs who are forced to use javascript even though they prefer python 2. They would write a different article for skilled js developers.

I treat Npm packages like a JavaScript and PHP. There was a need and it was done. In the short run it is good enough. But in the long run it may become complicated. For example what about essencial packages that wont be maintained anymore? What about depending on millions of lines of code from unknown sources? What about single dependency that is npm? It is a risk too.

So what exactly is the problem? I see a lot of talk about dependency hell and node_modules bloat but I don't really hear much about what the actual downside is.

Is it performance (I haven't experienced it)? Is it taking up disk space (I have terabytes to spare)? Does it add complexity (I would argue it reduces complexity)?

What exactly is the problem?

People who sin shouldn't complain about winding up in hell.

I'm not buying the narrative of "js dependency hell what are they thinking" anymore.

Try installing a package for Rust. Or Go. Or ... any language, really.

I’m not sure whataboutism is the way to fix the well-known issues the JS ecosystem has in package management. When I’ve installed a Go package or Python package I haven’t ended up with sometimes hundreds or thousands of sub-dependencies. Some packages can be ridiculous, but nothing like I have seen running “npm i” for something that seems like it should be simple. I apologize for not having an example off-hand but this keeps coming up precisely because it is an issue unique to JS in this case, even if other languages have it to a degree.

Try Feathers, a "lightweight web-framework for creating real-time applications and REST APIs" clocking in at about 600 transitive dependencies. How do you audit this?

Link for the curious: https://github.com/feathersjs/feathers

I think they just don't audit nor review the code they depend on.

It's now whataboutism. It's finger pointing from a lot of people who develop in languages whose story is only marginally better.

As I wrote in a sibling comment, I have a project I haven't even started yet in Rust [1]. And it's measly 6 dependencies pull a total of 197.

[1] https://news.ycombinator.com/item?id=22841742

I took a look, this sounds more unique to Rust than Go or Python, but they all can do it to some degree. I have no hands on experience with Rust to really have any reasonable input on this so I don’t want to make an uninformed statement.

If what you say is true, then Rust is suffering the same fate.

197 dependencies is too many to trust.

Gatsby has 19k dependencies. I don't think I can find something like that for Rust.

The largest, and not exactly admirable, I can find is reqwest that drags in 97.

Whilst 100 is a huge number... It's an enormous gap from the many thousands.

Javascript doesn't have a standard library (something a lot of languages provide), and nodejs's built-in functions library are... lacking. Also, the culture of providing small packages, being around for longer, and so on and so forth...

I have a project I haven't even started yet in Rust. Three dependencies in Cargo.toml. They download and install 34 dependencies.

I've now added three more (from kube-rs readme). It's now 197 dependencies. And so on.

And that it is not even the worst I have seen. I think the worst offenders are development tools like webpack.

I have a small project that I work on from time to time which uses 5 libraries (react, a map library and a chart library), typescript, and react-scripts (which I guess pulls in webpack and all the rest).

This is what happens when I run a npm audit (I havent' touched it for a couple of months)

> found 38934 vulnerabilities (38916 low, 18 moderate) in 906346 scanned packages

> run `npm audit fix` to fix 38620 of them.

> 314 vulnerabilities require manual review. See the full report for details.

While the real numbers are probably lower because there is a lot of duplication inside the node_modules folder, I find this ... astounding.

There are bad packages in rust, go, python (1), even C. The worst of these are closer to the norm for JavaScript though. Personally I don’t feel comfortable running JavaScript packages outside of some kind of sandbox (usually a VM) on my personal computers.

(1) in fact, seeing some of the worst kept me away from python for far too long until I realized plenty of python software doesn’t even need pypi and most only need it for a couple things with no transitive dependencies

Yes. Using third-party code is not a binary decision between “write everything yourself” and “use all the things”, it is part of the ongoing conversation about optimization: Development time, deployable size, build time, tool support, community support, training, documentation, etc.

If you mean that javascript gets a disproportional amount of hate, that might be because of how much more it is being used. It is true that rust seems to be ignoring the sins of the past and diving right in to having lots of dependencies, with the solution being to manage them instead of weeding them out in some way.

I agree that this isn't totally specific to JavaScript. It's just that the ecosystem is very extensive and so there are a lot of packages, because you don't need to write your own code for a lot of existing problems.

There is perhaps a something about the JavaScript audience tending to attract less advanced implementers. So you end up with a lot of otherwise considered trivial and low-quality utilities being published and pulled down from the registry.

Rust and Go tend to be for more advanced scenarios where things like performance, security and support matters tremendously.

Sometimes I feel like I am becoming increasingly out-of-touch with reality with how far this NPM/JS ecosystem nonsense has been taken. Is anyone still using this crap for de novo business applications that have real-world consequences? I have no problem if you want to build fantastical, sprawling node implementations for fun and potentially personal profit, but when your decisions start to involve other professionals who are simply trying to get the features implemented so they can go home...

I feel like we need to really think about the consequences of making these kinds of technical choices, especially in 2020 given all the horror stories circulating. Someone is eventually going to have to come in after the fact and scrape all the gore off the walls and redo your webapp in .NET/Rust/Go/etc. Why not start there and be done with it? You could save so much frustration. Is it about subjective/artistic preferences? At what point should the business owner start getting involved in these technical decisions? If the business owner found out their webapp was trapped in hell because you had a personal aesthetic attraction to one language over another much more pragmatic language, how do you think they would respond?

Full page POST reloads on each data submission aren't going to fly with most users, even business users. Neither is reloading a page to get new data.

My personal feeling about CRUD is this, you don't have to get fully on the crazy train and spend all your time with packages and build tools to achieve a nice product. Yarn is better for me. Very sparing package use. And a little light Vue with server side templating. No I don't need you managing routes there JS framework, that never was a good idea in my estimation. (Or maybe it is and I don't get it but I like incremental changes and simplicity).

Once the dust settles perhaps I'll move on to a more complex setup but I don't have time for it now. I'm trying to get stuff done. However this doesn't have to mean eliminating modern behavior or appearance. Which means some JavaScript. It's going to be required in general by end users.

Fwiw I've found turbolinks gives me enough of an SPA feel that POST reloads aren't a huge problem. If we need a very dynamic experience then Vue would-be my go to tool,but for the other 90% this is fine.

Reply on news.ycombinator.com