Hacker News Reader

Comment by thenewwazoo | Hacker News Reader

The npm team's lack of responsible action here is astonishing, if not surprising. If a token for these very popular packages was compromised, then every user of those packages could conceivably have had their tokens compromised, not just those that have it as a downstream dependency. That means every single package published between the vulnerable packages and the mass token invalidation is suspect and should be unpublished until it can be audited. npm's position that "a very small number" of packages were compromised is minimizing a potentially massive issue. Their promise to "[conduct] a deep audit of all the packages in the Registry" both misses the point and is ineffective.

I have come to expect no less from npm, and by extension, nodejs.

View on HN

NPM revoked all tokens. How is that irresponsible?

https://status.npmjs.org/incidents/dn7c1fgrr7ng

A malicious change could have already been published somewhere else, couldn't it have? And we just haven't found it yet?

Just saying they didn't do nothing. Maybe they didn't do as well as they could have, but they did do something.

Calling it resolved is worse than doing nothing. If they had done nothing, at least people would know that "If I run npm install now, that's bad". Now they've claimed it is resolved, which tells their users "It's okay to start installing things again" when it isn't safe until an audit has been completed.

For one, they've marked the issue as resolved before they've completed any forensic analysis to discover if additional compromised packages were uploaded with stolen credentials during the window in which they weren't revoked.

This gives the false sense that its safe to install packages again.

And the fact they don't get this is incredibly disconcerting.

There's also no systemic fix for this issue. Next week, the same thing could happen again.

It's time to enforce two-factor authentication for publishing packages.

It's time to publish a roadmap towards package signing and verification.

It's time to talk about sandboxing the install scripts to prevent token theft.

We're done for the day is so far from sufficient.

> package signing and verification

You're the first person I've seen mention this, which seems like it should be the first and most obvious line of defense against bad stuff like this. +1

People mention most of these steps every time npm fucks up (which is often).

Package management is a mostly solved problem, but npm refuses to learn from the 30+ years of experience that Linux and other communities have had.

I used to be one of the biggest HN proponents of dunking on Node. I would give it a thrashing second only to the windmill dunks I would line up on Go.

These days, with ES6, TypeScript, and React...it's actually all pretty nice.

But NPM is still and forever a mess that just plain didn't learn from its predecessors--and my only guess as to the cavalier treatment of security issues is that the NPM crew's a bunch of premillenial dispensationalists banking on getting raptured (or bought, I guess, but that's way less fun) before the chickens come home to roost, 'cause just about anything else seems implausible.

I am thinking about getting in on Deno only and strictly because maybe that can have a package management solution that is not held captive by an irresponsible venture-backed entity. It's not like anyone's breaking NPM's grip anytime soon. (Even if you tried, their API is so awful as to be functionally incompatible unless you literally just copy their internal structure, so...yeah. Great. Hooray.)

> ES6, Typescript, React

NodeJS didn't make those - they'd be around with any nodejs alternative.

btw, nodejs should provide some "isolated" mode (ie run as user "nobody-projectName-userName" - eg. "nobody-react-whatcanthisbee") and do some appropriate group permissions.

basic linux permissions can do a lot...

what saddens me is that there seems to be no way out. There are so many projects that worked fine in the past and now the're using npm, for example PhoneGap.

Signing packages works well for a linux distro. It's much less clear that it would help for a language package manager: https://caremad.io/posts/2013/07/packaging-signing-not-holy-....

None of the criticisms of package signing here apply in this case. Package signing does cover this exact issue, which we should describe to be specific:

(1) someone has published a package. (2) that package has become well known and used enough that the community as a whole trusts that the entity publishing that package is not there for malicious purposes (3) the maintainer of that package has their account compromised, either by credential leak or by a vulnerability in the package management system itself (in this case, the former and not the latter)

If the package were signed, then the attacker could not have published this fake package that lead to issues today. Does this solve the root chain of trust issue for whose packages you should trust? No, but nothing other than thoroughly reviewing every line can do that. Does it prevent you from having a random unknown person masquerade as the maintainer and publish a malicious package? Yes.

There are real additional trust issues to solve, but let's not let those detract from the fact that package signing would have prevented the exact issue which we saw today. Defense in depth, always.

I'm not going to say that you're wrong in any particular thing you said. You are correct that package signing would have prevented this exact issue we saw today, and I'm not opposed to an argument for defense in depth.

However, I am going to reiterate my original point with a bit of clarification: "it is not clear that package signing will prevent malicious actors from compromising systems using a language package manager with anything approaching the success of distro package managers".

I think what you're proposing amounts to a two-tier system. There is somehow a set of known signatures that are trusted by the npm consumer[0] and then a vast sea of untrusted signatures. Developers sometimes add libraries, and add one (or more?) signatures to their trusted set.

First, in the specific case of NPM, we have an existing system with huge numbers of transitive dependencies and no existing package signing. I don't see how you retrofit signing on to that system. Too many developers will ignore it, but also because there are so many libraries, you'll have an absolutely massive number of trusted keys.

Second, suppose you're starting from scratch, so you can enforce signatures from day one. You still have the problem of transitive dependencies. It's certainly more work for a malicious actor to either a) create legitimate libraries that will be included in other libraries, then convert them to malware, or b) steal a developer's signing keys, but neither one is remotely impossible, given the large number of libraries involved. You just need one part of that web of libraries to involve a sloppy or malicious developer, and you are hosed.

[0] Interesting question: how is this handled? Is it part of the lockfile for a project? Something system level, with the problems of devs installing random shit on their machines and ending up with too much being trusted, and also lack of reproducible builds?

That's from the maintainer of pypi. Pypi shipped SSH Decorator with malware that stole SSH keys. Another case that would have been entirely preventable with package signing. I've concluded they don't know what they're doing either.

I'm not surprised if you don't know about it, because it was [dead] on HN as soon as it was posted... which should tell you a bit about the echo chamber you're in here.

https://imgur.com/gdUFToP

Except none of these comments take into account the fact that actually no one verifies apt package signatures either, and this fairy tale world where all of this is a solved problem in some other domain is not manifest.

https://blog.packagecloud.io/eng/2018/02/21/attacks-against-...

The reason that package signing never really matters that much is that once you boil the thrat model down to package publish credentials are compromised or package repository infrastructure is compromised, the form of credentials involved is of little consequence to the prior and uninvolved in the latter. The threat is against the client, not intermediates.

The original developer here reused credentials. There is nothing in signing that protects from this attitude. This attitude is the one that also reuses credentials for signing keys, if encrypting them at all - I'd bet this user has numerous stale ssh keys and never encrypted any of the secrets. Some of the top eslint contributors have multiple short rsa keys on their GitHub. None of them have modern keys.

There are more effective places to invest to better protect users. Auditing infrastructure for example.

The most worrisome on that page is the arbitrary package attack, and on my Debian installation it's not feasible. The insecure apt.conf settings are not enabled on Debian by default. Saying that the package signing of apt is absolutely ineffective is dishonest.

Note that metadata signing in apt is just indirect package signing. The package sha256 sums are part of the metadata. It looks like that dpkg too have support for package signing, but at that point it would be redundant.

This is an obvious ad hominem attack, and completely missed the point of the pypi post.

If someone says "this problem cannot reasonably be solved", you don't get to discredit them by saying "look, you had the problem!" You have to actually rebut their arguments and say that it can be solved.

It would have prevented this from happening.

In fact, there's a pull request from 2013 for GPG package verification. It took over a year for a response from NPM, shooting it down. There's already an "I told you so" in the thread.[0]

>Thank you for your time and effort.

Yeah, thanks to you too, NPM. No time or effort to go around for progress on this issue in the interim 3 years, apparently.

[0]https://github.com/npm/npm/pull/4016#issuecomment-76316744

If the hacker has your password, don't they also have the ability to publish a public key used to verify the signed package? It presumably would protect against distribution of a fake package outside of NPM, but if your NPM account is hosed isn't it too late?

If 2fa is enforced, having your password doesn't get you anything. Publishing an npm package isn't having a Twitter account, it's one of those things where enforcing 2fa shouldn't even be a usability question.

how does 2fa work with auto publishing CI pipelines?

Not necessarily. Have NPM store the public key, and require a lot of red tape and time to update it (versioning with delays).

Require multiple keys to sign or vouch for a package before publishing is complete (log of reverse dependencies +1 maybe).

There are lots of options.

I don't think sandboxing install scripts will help much, as any code in the package will be executed when that package is require()'d. You really need to sandbox the whole of `node`.

* eslint gets compromised, and 3.7.2 is published

* eslint user FooCorp also gets compromised, and a similarly-malicious version of foolib-js gets published that includes the _same code_ to steal tokens

* npm invalidates all tokens

* you decide to use foolib-js, and your newly-minted token is now compromised

npm are fucking this up, and royally.

While this is possible, I'm willing to give the NPM team at least a little benefit of the doubt that they actually researched the access logs before they state this:

> We determined that access tokens for approximately 4,500 accounts could have been obtained before we acted to close this vulnerability. However, we have not found evidence that any tokens were actually obtained or used to access any npmjs.com account during this window.[1]

I get that it's possible that other modules could already be infected, but it's also true that other modules could have been similarly infected long before this one.

[1] https://blog.npmjs.org/post/175824896885/incident-report-npm...

Your quote wonderfully illustrates that npm are either being obfuscatory or entirely missing the point.

How did they determine tokens for 4,500 accounts could have been obtained, and what is that even supposed to mean? The problem here is that any user of these packages could have had their .npmjs file read and exfiltrated, not just some upstream package maintainer. Were there only 4,500 valid npm tokens or something? I cannot imagine that is the case.

So either they looked at 4,500 packages uploaded during the compromise window and they're not explaining how they undertook to do that, or they don't understand the vector and are minimizing the severity of the issue.

I would assume their logs would possibly tell them which tokens were associated with the users that downloaded v3.7.2. npm probably doesn't need credentials to download a package so the number of downloads is likely higher. Determining other packages affected are another matter entirely and no one can say this attack vector is only bound by this specific date window. This could've been way more widespread unless they're unpacking payloads and grepping for key pieces of this specific attack.

I think it would be helpful if they could expose some of those logs but considering the meat of what matters would be the IP addresses to verify if your machine was compromised (or your CI server) that GPDR effectively wiped that possibility off the table. It would almost behoove them to setup a kind of haveibeenpwned service where you can check against stuff like this in the future. It's not like this can't happen again as the hole hasn't been closed completely, only this one set of compromised packages appears clean for now.

You can see a list of all npm packages uploaded between the time that the first compromised eslint package was uploaded and the token invalidation time here:

https://gist.github.com/thenewwazoo/0306aa06aafe7807497ed1db...

To follow up, I've downloaded all of the versions indicated here and none include the strings `raw/XLeVP82h`, `sstatic1.histats.com`, nor `statcounter`, and none of the included `build.js` files contain `eval`. Not an exhaustive test.

Further follow up: I've updated the gist to include all packages between the time at which the first compromised package was uploaded and the time at which the keys were invalidated (which was later than the invalidation threshold). I'm working on auditing the larger (~2000) list of packages now.

Final follow-up: a more-exhaustive search also returned no evals in build.js files, nor any obviously-suspicious strings.

Have you checked for

    \u0065val

    e\u0076al

or any other combination of it?

This is impossible in general.

  eval === this[[1,18,-3,8].map(x=>String.fromCharCode(x+100)).join("")]

https://jsfiddle.net/hkvu9s47/

That's the point. That just checking for occurrences of strings like 'eval' just gives you false sense of security.

> That means every single package published between the vulnerable packages and the mass token invalidation is suspect and should be unpublished until it can be audited.

I wonder how many packages in fact got published during that nine-hour time window. Does anyone know what kind of number that would be?

I'm currently dumping the npm db right now to figure that out.

eslint-scope 3.7.2 was published on 2018-07-12T10:40:00.478Z eslint-config-eslint 5.0.2 was published on 2018-07-12T09:49:24.957Z tokens were invalidated at 2018-07-12 12:30 UTC

> tokens were invalidated at 2018-07-12 12:30 UTC

I'm reading it as tokens created before 2018-07-12 12:30 UTC were invalidated, but the invalidation itself happened on 2018-07-12 18:42 UTC according to eslint's post-mortem[1]. I think the latter is the more relevant date.

[1] https://eslint.org/blog/2018/07/postmortem-for-malicious-pac...

If I understand the npm incident report correctly, the answer is none:

Source: https://blog.npmjs.org/post/175824896885/incident-report-npm...

The wording is pretty vague, I guess intentionally, but saying "we found no evidence..." isn't as confidence-inspiring as if they'd said "we determined that no...".

  "npm has revoked all access tokens issued before 2018-07-12
  12:30 UTC. As a result, all access tokens compromised by
  this attack should no longer be usable."

"access tokens compromised by this attack should no longer be useable", but tokens compromised by other compromised packages published during the intervening period will still compromise you.

Reply on news.ycombinator.com