Worrying about the NPM Ecosystem

The npm ecosystem needs something like the distinction between Ubuntu's "main" and "universe" repositories, so that you have a smaller subset of known-good packages with harmonized transitive dependencies, stronger centralized curation, a lot more direct scrutiny, tighter control over versioning, and some party that is responsible for addressing vulnerabilities and ensuring appropriate maintainership.

As a venture-backed startup, the company behind npm needed the number of packages and the number of downloads to constantly trend upward in order to justify their valuation. This led to extremely poor policy and prevented them from taking strong steps to remedy the deterioration of the ecosystem. Now that it's owned by Microsoft, there's an opportunity to fix this.

Linux distributions have decades of experience solving these problems, there's no excuse for the JavaScript community to continue ignoring the longstanding precedents and best practices that have emerged from that experience.

This post describes several potential "problems" with NPM packages, but not _why_ they are problems in the first place. The author doesn't say why they consider circular dependencies bad, or why dependency depth should be low, or even why their categorization of packages is the way it is (and leaves out large categories like tools and CLIs).

I don't think that a high number of dependencies is necessarily bad, or a low number is necessarily good. A dependency can often have factored out common functionality into a shared and higher-quality project than the same bespoke implementation re-implemented in several packages.

I think we'd need some more complicated metrics, possibly looking a bit like a web of trust analysis, to determine quality and health of packages. Depending on a well-maintained, popular, and highly-trusted package like lodash or express shouldn't be an indicator of low-quality or low-trust. We also have deprecation and security notices provided by NPM.

So I'd be much more interested in questions like "how many unique, low-trust packages are included in this package graph?" or "how out of date are the dependencies?" to try to estimate package health. From there I'd be curious to know how healthy the ecosystem is in aggregate, weighted for usage.

One of the author’s points it the thing that makes me crazy about the JS ecosystem “... find a package ... it might depend on hundreds of other packages, with dependency trees stretching ten or more levels deep...”

I cannot count the number of times when I’ve installed a package and got >500 or >1000 packages as dependencies.

Just today I installed ‘alptail’ a simple set of components using AlpineJS and TailwindCSS (I love both!) but it npm installed 1314 packages! For a simple set of maybe a dozen components.

This is madness.

How do we make this better?

Maybe we need more support from tooling to understand just what is being used down the tree of trees of trees?

Maybe as lib writers we should seek to implement our own supporting functions (within limits, of course) rather than having a first reaction “let’s go grab XYZ for that” and inherit XYZ’s entire dep tree.

Not sure what the answer is but what we have today is a beautiful mess (love/hate it).

Thanks for letting me vent a bit. I feel better. But this really bothers me.

The chance to fix npm is dwindling quickly. With each year, a culture sets in that will get harder and harder to fix with time.

Personally I suspect that the lefpad debacle might have been the point of no return for npm; the fact that such a simple package managed to break everything g should have been a wake up call, but it does not appear that the community fixed much.

My non-technical thoughts on the npm ecosystem:

1. Split package-lock into a human readable file and a machine readable one. Right now it's just pain.

2. TRUSTED PACKAGES. Somebody has to assign trust. Companies or highly involved developers should become curators.

3. LIMITED Packages with reduced security risk: File Access, Network Access, package access, OS access, etc...

4. Transit npm package best-practices into a single module format. Reduce clutter and number of files in node_modules.

5. Better tagging. Allow votes on tags.

6. Add monetization links for package authors to npm (blunt) and to the CLI (subtle, e.g. 'npm pay leftpad' opens the monetarization page on npm.).

Additional ideas:

A: Buyable certificates/insurance for packages.

B: Better clone, modify & re-publish workflow

C: Integrate FAQ/Forum directly in NPM directory website. Simple downvotes/upvotes for visibility. Allow and ask for readme pull requests from npm package page.

Hate to break it to the author, but npm has been a raging dumpster-fire for over 4 years now. My personal advice on how to live with it:

- Assume you will have to rewrite your app in a year or less from now and organize it as such. This won't be easy when dealing with release deadlines which result in all-nighters full of sphagetti code. Whether you're doing react, vue or angular, just keep your components and logic small and if you need a package, (say for handling CSVs) then look for one that promotes itself as having no dependencies. Keep in mind that while re-writes can be a taboo subject to many managers, they can actually be the fastest way to solve deep dependency issues.

- Unless your framework requires it, try not to lock your package versions, this is a very common bad practice in my opinion. ```npm audit``` is your new best friend, make updating to latest part of every sprint. If you're just 3-4 months out of date these days you're likely to have over 40 security vulnerabilities. If you're 6 months out of date on a public facing application and need to pass a security audit, you're not gonna pass it. Compliance requirements, like SOC2 or healthcare ones are getting a lot more thorough as well. It's a lot less work to update continuously.

-Secure your devops. Any npm package that runs has full acess to your entire disk and network I/O. The cheapest way to harden security is to move your devops to aws or gcp. They can afford to audit their networks and OSes a lot more often than you can. If you're doing it alone or on a very small team, i'd recommend trying out aws-amplify or firebase hosting on gcp.

-If you're doing node on the backend, go full typescript. I think it's easier to adopt a "strongly typed" mentality when your logic is clearly defined. If you don't feel confident in your use of typescript, then don't use it in the front end. You're gonna have a bad time, especially with state management in React. However it is getting more mature and package managers are learning to avoid type issues too when updating their methods.

- And last, when picking a new framework or boilerplate or package, run npm audit on it first then make sure to read up on their github issues as much as you possibly can. It's very important to get not just a technical feel but a social one too on how their devs react to their issues.

A lot of the results of this analysis are a consequence of actually good UX in dependency management.

In NPM publishing a library is really easy. To compare with java, I recently had to follow a [three part medium article](https://proandroiddev.com/publishing-a-maven-artifact-3-3-st...) to publish. NPM: one login, one cli command and you're done. That lowers the bar to create a library, there are a lot of low quality libraries, but there's also much more diversity and really cool libraries. In java I've seen a lot of copy pasting. In Node you just create a library. You can worry a lot about 1k dependencies, but a lot of those will be one liners.

In node, it's basically impossible to have conflicts with dependencies. Again if we compare to Java. In Java namespaces are global, if there are two libraries sharing the same package name they will collide. That makes deep dependency trees actually a liability. I've seen recommendations of not using guava if you are writing a library, because they change the api so much that it will most certainly give problems if your user wants to use it as well, so much for a library. In Node, in contrast, if there are two different libraries that require different versions, they will each have their own dependency version. That has the cost of space, but there it saves hundreds of hours on conflict resolution. You can even have two different versions of the same library with different name, in case you need to partially migrate a project.

Making it easy to create and consume libraries has of course a clear consequence, more deep and big dependency trees.

The solution mentioned at the start of the article is what distribution package maintainers have been doing for decades. I would take this account as a giant flashing warning sign of what could happen to the flatpak/snap ecosystem, and also a flashing warning sign for languages that are developing their own package manager, especially if it is difficult for packages from the language package manager to be integrated into actual system level package managers. The curation effort required to get NPM into manageable shape is almost certainly beyond the scope of any traditional community effort.

For most other languages the package ecosystem is a competitive advantage; Perl had CPAN; Python famously comes with batteries included, but is used where its batteries don’t matter (numpy and friends, django); rubyists will go on about their gems for as long as you care to listen. JS is the only ecosystem I know of where the libraries are something that scares people away from the language. My own opinion is that npm is not state-of-the-art, and there is a culture problem (starting with wrapping single statements in a library — leftpad is most famous, but wrapping a single regex in its own, incompatible wrapper seems to be an accepted way to do things, too). The culture problem may be unsolvable.

Nice to see some actual numbers and research on the topic. Most complaints about NPM seem to fall into the category of fretful, emotional hand-wringing (although there's some of that here too).

If we measure success by the amount (not the percentage) of high-quality packages available, NPM is easily the best package management ecosystem yet. But there's definitely room for improvement.

One major improvement I'd like to see is some solution for unwanted indirect dependencies. If I depend on package A, and A provides features X and Y, and I only need it to do X, and the code for doing Y depends on package B, then it shouldn't download B. The recommended solution is to split up A into a family of packages (like lodash's per-method packages), but that just creates a mess of versioning that's annoying for both the authors and the consumers. And those massively-multipackage libraries (like Babel) just seem to create dependency hell situations that semantic versioning is not good enough at solving. So maybe instead of package splitting, the next-gen package manager could have first-class understanding of ES6 modules, with tree-shaking logic to prevent unused indirect dependencies.

I think that a lot of ideas here are really good, but they shouldn't be implemented in npm. Npm is not meant to be curated. NPM is meant to be a canonical way to reference a dependency, and that's about it.

I don't think it's NPMs role to assign "quality" to a package (even though ironically they have a quality metric) -- that's something a third-party should do on top of npm.

That's the unix philosophy in me speaking.

I think the problems articulated in this article will make Deno attractive to many JavaScript developers. The standard Deno library has everything needed for building server-side dynamic websites.

But I think what will speed adoption of Deno is something that has been little-commented on: the runtime is a single-executable. This massively simplifies deployment.

Contrast that with Node, Python, Ruby and the way they spew out dozen, often 100s, of tiny script files all over the place. Simple, uncomplicated deployment solutions (particularly for self-hosting) simply don't exist with many interpreted languages. Whatever you think of Deno, their decision to create a single executable runtime will make it attractive to many developers.

I have decided to use javascript strictly for frontend only, no more node.js or express or whatever else for backend. I will just use non-js backends these days, which are plenty.

It's sad I still have to use npm even for frontend development nowadays, along with babel, webpack and a matrix to get started.

I now stick with jquery(yes I know, but its concise syntax matters to me) and bootstrap, and they seem working for 99% use cases for me. super easy to get start and maintain, good enough for my use cases.

The only problem I see is the lack of a comprehensive standard library for Node.js.

A package depending on itself is just an accident (and probably something npm ought to detect on publish and reject, but alas...)

However, I believe a 1-package circular dependency is often the result of a developer mistakenly adding a package as a dependency when they mean to add it to `peerDepedencies`:

e.g. in TFA: "babel-core depends on babel-register, which depends on babel-core... yeoman-generator depends on yeoman-environment which depends on yeoman-generator."

Peer dependencies, of course, no longer exist because it got deprecated because raisins.

Which leaves us with zero options to define sideways dependencies.

"is this package 'healthy'?" is a good line of thought, but the proposed metrics are all less useful than just opening the source on Github or unpkg.com and giving the source code a quick skim read.

If it looks too scary for you to maintain, you shouldn't take it on as a dependency.

> It’s not really a technical problem, but mostly a social one.

This is absolutely correct.

The problem with the javascript ecosystem is that javascript ecosystem rejects the problems with the ecosystem.

Why why why is the default behavior of npm install to choose a range of versions rather than a single version.

I don’t want “a version compatible with lib 0.1.0” I want lib 0.1.0 ... package-lock is a strange bandaid over this.

If I want the code my app depends on to change, I’ll change it.

Btw, nobody is forcing anyone to use any packages from npm. You can easily use whatever repository you want. You can store your dependencies in your source repositories together with you application or another entirely separate repo.

JavaScript is a huge, perhaps the biggest, software ecosystem in the world. There is a vast amount of code out there and you are going to get some bad stuff if you are not careful. This does not mean it is bad at the core. It means that it comes at the cost of doing security work upfront if you care about this. while benefiting from the crowd-sourced experience of the masses.

While I agree that NPM can be improved, most people who are complaining about the number of dependencies have no idea what they are doing.

If you install a package without the --production, which is default, you get everything including perhaps some build packages like babel, linters, and what not. These dev dependencies tend to be huge. That is how it works. Similarly, if you install the dev version of a library in some other language you get not just the source but perhaps hundreds of megs more of other source. What is the problem?

If I install React it's because I trust React. If React installs 10000 packages, I see no problem because I assume that React knows what they're doing, otherwise I wouldn't trust them and I wouldn't have installed React in the first place.

If I don't like deep dependency trees, then I install packages that I trust not to use deep dependency trees (or rather graphs since they can have cycles).

What if npm packages are re-coded by popularity? In the sense that any package that has achieved high connectivity and popularity must eliminate as much dependencies with pure JS code. But again that would cause code bloating. Dan Abramov had a talk about abandoning Clean Code. It is kind of strange that simple React app now days has NPM folder with 1GB of files, but I guess we got used to.

I refuse to use npm at all. Too risky.

I only use libraries if they can be added as a first-order dependency (i.e. in a script tag on the page). This is still risky, but I can mitigate it somewhat. And at least I'm not giving a bunch of random maintainers commit-to-prod access in my core code.

Been looking for mdns browser package and still not sure how to filter out node packages on NPM.

Cyclical dependencies are a feature, not a bug. Many languages don't allow such a feature to exist at all with their compilation/interpreter models.