Announcing NPM 6 | Hacker News Reader

Announcing NPM 6

medium.com

104 pointstheodorejb51 comments

npm has eroded so much of my trust that I am hesitant to switch back to it (from yarn) any time soon.

I've tried npm out every few months (since npm 3), and have consistently run into infuriating bugs or unexpected behaviors.

Much of it has been fixed over time, but the frequency and duration of these issues is concerning—and, I think, points to architectural deficiencies being the root of the problem. For example:

* npm 5.0.0—5.7.0 didn't play nice with git-based dependencies (https://github.com/npm/npm/issues/17379)

* npm 5.0.0—5.4.1 edits package-lock.json unexpectedly (https://github.com/npm/npm/issues/17979)

* npm 5.0.0—5.4.? doesn't honor incompatible version differences in package.json compared to package-lock.json (https://github.com/npm/npm/issues/16866)

* Take a look at the issues labeled as [big-bug], and how long they've languished: https://github.com/npm/npm/issues?q=is%3Aissue+is%3Aopen+sor...

* and a bunch of others I can't remember off the top of my head; especially nondeterministic behavior in the v2 and v3 era.

---

Btw, I'm not trying to tear down npm contributors here; people have put in a monumental amount of work into the project, and the node ecosystem wouldn't be where it is without npm.

Another gem: Trying to use `npm shrinkwrap` will cause npm to enforce peer dependencies as required instead of just a warning https://github.com/npm/npm/issues/12909 Despite the issue being automatically closed by a bot (due to apparent lack of interest from the npm devs) this bug is still going strong a couple years later. Switched to yarn haven't looked back.

npm shrinkwrap never worked well for me whatever the version, I always had some random error at the end with it. I confirm having this bug now which makes it useless but even before it was something else. One time when it was still working, it was even producing invalid JSON I had to edit manually (I wonder how that's possible).

> Take a look at the issues labeled as [big-bug], ...

I'm amazed at the disparity between the number of issues marked [big-bug] (23) and the number of those that are in the big-bug-backlog milestone (4).

> Customers of our paid services will receive additional pre-publication vulnerability disclosures, formerly the NSP’s premium tier.

How common is early access to vulnerabilities as a paid service? This seems to have some ethical concerns since it dramatically increases the odds of weaponization prior to public disclosure.

don't paid npm customers get to run their own instances of npm? They need some kind of lead time purely from a technical point of view to patch.

I think the point was that malicious actor gets early access to "exploits" by becoming paid customer and has a lot of time to pwn general public.

This seems too little too late. Yarn is fast, stable, efficient, and (most importantly) reliable in getting things done. Doubtful anyone using yarn would switch back at this point.

Meh; unless you're doing something really complex or ran into an odd bug, npm is fine and so is yarn. They're both fast and stable enough and they both do _basically_ the same thing.

I don't understand the turf war; the node module ecosystem is incredibly shitty (good luck running any modern framework without pulling in hundreds of dependencies) and these two tools do basically the same thing. Just use whichever one works ¯\_(ツ)_/¯

Meh? NPM only started to get better after yarn was released so this is over several years of experience.

Yarn also solved major speed and reliability issues. I'm not sure why that keeps being glossed over but NPM often had problems with reproducible builds and generally keeping things working. Just check the top comment on this page to see the outstanding bugs.

Dude, I can't even begin to start counting how many times I have had to rm -rf the node_modules/ folder and do an npm install afresh, just because some dependency has screwed up the other.

yarn has given me peace that I did not think was possible.

YMMV but in my experience npm would break consistently doing the most trivial of tasks. Switched to yarn and haven't once looked back.

    npm@6 is up to 17x faster than the npm of one year ago.

Some advice for other people doing performance tuning:

There comes a point where impressive gains cross over from a brag into an admission of guilt. The long slow is a big part of why yarn exists.

I don't remember it too clearly now but a few years ago there was a case of "One line of PHP changed and composer is now 70% faster". I'm not comparing anything, just got reminded of it.

It happens, just not usually like npm.

The worst performance bug I ever reported on a piece of software had an O(n^5) runtime. But our n was about 5x what anyone else was doing so it went from a couple minutes to all day. Luckily they found a workaround, because that was brutal.

We switched back months ago. We tried yarn for a few months and while there nothing wrong with it, after npm 5 came out there wasn’t a compelling reason to keep using it.

What was the compelling reason to switch back to npm versus yarn?

Everyone has NPM. A subset of them are familiar with Yarn. So it makes sense to use NPM if Yarn doesn't confer any operational advantages that you care about.

That makes sense, thanks.

For me, the reason was, why add an extra layer when there isn't much of a reason to do so. All packages are at least tested on npm.

That's definitely a fair reason, thanks.

Integrating nsp is a very welcome move. It's heartening to see npm taking the security issue seriously and taking steps to start addressing the problem.

That said, the problem remains largely cultural rather than technical. The community needs to make significant changes to how it consumes and publishes dependencies in order to reduce the attack surface. The massive, bloated dependency graphs that developers who use Node tolerate with cavalier indifference are not safe or sustainable in the long term.

Hopefully increased awareness of insecure dependencies among users will increase reporting of them to dependant projects, which will encourage some or all of the following behaviours:

1) projects to keep their dependencies up-to-date

2) projects to use less dependencies

3) consuming projects to switch their dependencies to ones that do the above

4) users to switch to projects that do the above

It would be interesting to track the average number of dependencies (total number in dependencies in the graph) over time to see how it is changing.

Those performance improvements sound too good to be true, how does this compare to Yarn? Although these are all excellent features I am disappointed to see that there is no support for Yarn's "flat mode" which installs all dependencies in node_modules (and not endlessly nesting them).

They probably found some silly anti-pattern they were doing previously that was killing performance for no good reason.

The entire history of npm screams "we're pretty much learning all of this as we go, please bear with us".

While there's nothing wrong in that (people have to learn somehow) it does make it very, very hard to put any trust in that project. Especially when said project is at the very center of an entire ecosystem, it makes it hard to trust the ecosystem as well. I'm already seeing a lot of good developers move away from node as they gain experience and realize things should be much better.

Its pretty much standard for literally any project I've seen without senior devs to see these bugs before a release.

> I'm already seeing a lot of good developers move away from node as they gain experience and realize things should be much better.

What are you seeing them move to?

Mostly Go but some Rust and Clojure as well.

I converted a lot of developers to Clojure myself. That language is by far the most fun I've had writing software in almost two decades of experience, where the fun comes from having simple yet really powerful tools rather than easy-to-learn-but-generates-complexity ones.

TypeScript seems to have been attracting some srs engineering talent into the ecosystem. If your backend needs fit into the node.js box, TypeScript is pretty compelling as unifying front/back end language; at least as a starter pack.

I used TypeScripts for a few years in the past. That was before knowing about ClojureScript however :)

Having done projects in both languages the later is definitely much simpler and this translates into less code that does more. Its then much easier to maintain and add features.

Most of the JS/TS projects I see lose iteration speed over time, while CLJ/CLJS projects I see gain iteration speed over time.

> npm@6 is up to 17x faster than the npm of one year ago.

I think by "npm of one year ago" they mean npm 4, since npm 5 came out last May. If so, "up to 17x" is plausible. But even though technically true, this is a cheap marketing move, IMO. If one doesn't read carefully, it may register as "npm 6 is 17x faster than whatever came out last year, that is npm 5".

How does it differ from the flat dependency support added in NPM 3?

The perf improvents are real, I recently updated our build tools from npm3 to npm5 and it shaved minutes off. npm ci is a real treat as is reliable parallelism of npm i.

IMO yarn was a protest movement when npm wasn't adding the features devs wanted. npm5 is the direct result and yarn should pat itself on the back and bow out to avoid fragmenting the community.

You make it sound like yarn was some sort of community a fork to plug some gaps.

Why should yarn bow out? Correct me if I'm wrong, but the yarn project didn't do the following: - put a pre-release build under a tag that looks like a normal build without a pre suffix - had documentation that expressed sudo as an acceptable way to do things to end users - then had a bug which torched machines if they ran any npm command under sudo. (which you shouldn't do anyway, but see point 2) - when pushing the fix, put hashtags in the announcements with a distinct tone of blame towards the end users.

> then had a bug which torched machines if they ran any npm command under sudo. (which you shouldn't do anyway, but see point 2)

This was the famous "rm -Rf /" bug.

> put hashtags in the announcements with a distinct tone of blame towards the end users.

What are you referring to?

> What are you referring to?

Probably this: https://blog.npmjs.org/post/171169301000/v571

Wow, that's very immature.

> bow out to avoid fragmenting the community.

No thanks. NPM caused the usability and feature problems first and there's no reason it won't happen again if there was no competition. Yarn is still better overall and I'm much rather they stick around.

Yarn feels more stable to me. I don't know how many breaking changes they've made but it sure feels like fewer (the version 1.x shows that).

Also, I think npm released it's lockfile too quickly. It didn't work for quite a while. Maybe it does now, I don't know, because any npm projects I have I have turned off package-lock due to it not working in the past. Yarn's lockfile just works, and always has.

So I'll stick with Yarn as long as they just keep it stable the way it is.

>In this winter’s ecosystem survey, we learned that 97% of worldwide JavaScript developers rely on open source code in our projects.

What runtime do the other 3% use to evaluate their JavaScript? Are they not using npm? If so, why are they responding to a survey by npm?

I am very confident that >3% of respondents did not understand the question.

Yeah, that’s lower than the Lizardman’s Constant*

* http://slatestarcodex.com/2013/04/12/noisy-poll-results-and-...

I was doing some statistics on a gaming site, and thought the average age of the users where surprisingly high, but not so much off I suspected a bug. I did investigate anyway and it turned out some users put 9999999 as their age, skewing the average. There's always noise, but I'm against filtering it out, because often it's in the outliers where the interesting stuff reside, and when it comes to software there's usually where the bugs hide. As for the average age I did the avg on those with age between 0 and 100, as I also found out it was possible to enter a negative age.

As I understand either they don't use open source projects, or (admittedly less likely) they're using an alternative serverside JavaScript runtime such as TeaJS/v8cgi, RingoJS, Helma, etc.

CommonJS/SSJS/JSGI/require() on which the node.js API is based started as a cross-platform effort of those. Maybe something like it will become relevant once again with polyglot runtimes a la GraalVM.

I built a runtime comparison of npm and yarn a while ago. It builds the same two things every day with different configurations (cold cache, installed, package-lock file)

The data can be seen here: https://docs.google.com/spreadsheets/d/1ZE5B4qJw1kNGMzjgslcW...

Am I reading something wrong or is the comparison really indicating that yarn is several times slower than npm?

The general conclusion and our experience has been the opposite and one of the big reasons why yarn exists.

Do those graphs show time in seconds? If so, I'm shocked by how slow yarn is...

I'm very curious what will they do when Python 2 reaches EOL. Will they keep depending on deprecated software?

There is no way for npm to guarantee security, other than auditing each version from each package one by one. Then, what is secure for one user may not be secure enough for other.

So it is misleading to say it is secure and that security is built in.

Most likely only popular packages, forget about making `npm publish` a free security review (unless it runs custom eslint rules for stuff like Regex backtracking, etc.).

And probably only guarantee it for particular version.

And "guarantee" is probably too strong a word for it, unless there is a contract with some kind of liability attached.

Same for "security", from the wording of it, they promise notifying about vulnerabilities, not performing comprehensive audits.

> So it is misleading to say it is secure and that security is built in.

Well, it is a PR blogpost with literally that goal of wording most basic and boring things in a way that sounds maximally sensational and groundbreaking without becoming a lie.

Example: see how they comment on performance improvement. Instead of "compared to previous major relase, npm 5" it says "compared to 1 year ago".

Yeah, hopefully now they don't have a progress bar that consumes most of the running time :)

Reply on news.ycombinator.com