Embedded Malware in Coa | Hacker News Reader

Embedded Malware in Coa

github.com

137 pointsStevePlea84 comments

For anyone interested, the malicious code can be found in the following link:

https://github.com/veged/coa/issues/99#issuecomment-96153687...

TLDR: The attacker injected an attack code as coa's `preinstall` script, which executes an obscurely-named file ("compile.bat"). This file is fully obfuscated, but what it does is basically to pull exploit DLLs from the attacker's server and install 'em.

I think the fortunate part of this accident is that the attacker failed to deploy the malware in his/her first attempt; v2.0.3 only contained the half of the changeset that the exploit needs to work (which accidentally broke tons of CI builds); So some developers could notice that something is wrong a bit early.

ahhh nice, it's been years since i've seen a obfuscated .bat!

very nice use of substring, but a bit too linear... with some input redirects and nested spaced variables it would have become more robust and unpredictable, but i suppose nowadays batch chiselers are rare.

edit: by the way in the article is missing a -useless- decoded line (n.4)

Earlier post (https://news.ycombinator.com/item?id=29111279).

The thing that should be causing concern is not so much these very loud obvious attacks, but how many better attacks that are harder to detect, are currently happening.

With 1.7M packages and an ecosystem that favours lots of 3rd party package usage, NPM is a large target. Whilst NPM isn't the only repository to have this kind of issue, it's definitely the largest attack surface.

Reminder that people should seriously consider disabling the install-scripts.

Personal system-wide config:

     npm/yarn config set ignore-scripts true -g

and add & commit a .npmrc/.yarnrc file with

     ignore-scripts true

Yes, this will cause headaches in some (increasingly rare) cases where some package actually needs those scripts. You can fix this with custom install scripts that take care of running install for those specific packages.

And yes yes, as people love to point out, this isn't exactly a bulletproof solution either. The attacker could just put the malicious code inside the package's code and wait for it to be actually executed. But again and again, they don't, they choose to use the package's install scripts as the place to do their dirty work.

So in practice this policy would've alrady protected you from who knows how many of these attacks, and my guess is that it'll continue to do so.

I think NPM should consider flipping the default on this. Code that requires an install script should be the odd case that draws scrutiny.

or, cause enabled is the default right now, it's way easier to spot malicious packages right now?

I disagree. We don't know how many such packages run installation scripts without noticeably breaking.

You can find out which packages require postinstall scripts and then run them manually:

  $ grep postinstall node_modules/*/package.json
  node_modules/esbuild/package.json:    "postinstall": "node install.js"
  $ cd node_modules/esbuild
  $ npm run postinstall

And to clarify, you should run this once by hand to collect the list, don't add the grepping to the actuall installation script or you are back to square one :D

Here's what I use with yarn, install.sh:

    #!/usr/bin/env bash

    yarn

    function run_install() {
      local dependency="node_modules/$1"
      if [[ -d "$dependency" ]]; then
        yarn --cwd "$dependency" run postinstall
      fi
    }

    run_install 'esbuild'
    run_install 'other_package'
    ...

Good point and nice script. I am going to use it in my projects.

I wish package.json had an option where I could explicitly mark which packages can run postinstall scripts.

> explicitly mark which packages can run postinstall scripts

Here's an RFC on exactly that: https://github.com/npm/rfcs/discussions/80

NB: Setting “ignore-scripts” globally means you can’t even “npm run” scripts manually, for packages you’re authoring (or at least that appears to be happening for me).

Edit: Actually, on NPM 8.1.0, setting "ignore-scripts" globally still allows you to run scripts manually with "npm run".

The behavior I described (where it just blanket ignores ALL scripts, even ones you try to run) existed on NPM 6 (which is what I was using until a few minutes ago). I couldn't find the exact version where this behavior was fixed.

A separate advisory says the npm package "rc" is also compromised. That's also a highly popular one, according to the npmjs stats (1,323 dependents; 14.2 million weekly downloads).

https://github.com/advisories/GHSA-g2q5-5433-rhrf (" Embedded malware in rc" "critical severity")

Notable that both advisories link to the virustotal entry for the same file hash (same malware).

@dang Could the title be updated to include the names of other affected packages?

For anyone else wondering, Coa is a CLI arg parser for Node.

This makes me appreciate Deno's focus on security. Having things like file and network access 'opt in' seems like a no brainer when we see how easy it is to simply install an npm package and find yourself vulnerable to malware.

Any protection offered by Deno isn't robust. Only OS-level protection is secure. Any tooling will require access to file system - and file system access is enough to compromise developer system.

Let's assume that you have Deno compiler for other language. You run it through seemingly innocent deno run "https://..." --allow-write=. src/ (you use optional parameter to --allow-write, right?).

Unfortunately, webpage hosting script was compromised. Now our compiler can write to .git/hooks, .npmrc (npm can do arbitrary script execution in version 6 or lower, even on npm --version), .idea/ etc.

Sorry, but no. This should not rely on an individual, it should rely on a bullet proof process.

Are other languages/runtimes also that risky as Node with npm?

npm packages seem like a cardhouse.

I know that the node_modules folder is often times criticized for its sheer amount of 3rd party libraries. Is it because of JavaScripts "missing" standard library?

I will be downvoted to hell for saying this. But javascript ecosystem is where most newbies come. (Low barrier to entry and it also seems hip). With no regards to security, maintainability or reliability, fashion chasing blog-happy hipsters.

The amount of churn in JS ecosystem, security incidents like this and general crappiness of websites can be generally explained by how immature these hipsters are.

PHP has a similarly low barrier to entry, but doesn't seem to suffer as much as JS.

Perhaps because dependencies are more curated in PHP due to clusters of dominant frameworks, rather than a proliferation of smaller libraries.

Not disagreeing with organization around frameworks in PHP, but apparently PHP suffered a lot from low barrier, in terms of security especially. That was the time package management wasn't that widespread yet, which IMO limited this kind of stuff. But there were, for sure, many applications where PHP could be blamed for security incidents and general unreliability. PHP improved quite well though.

Now PHP isn't hip anymore, node js is super popular hip thing, and every tom dick and harry from art school in US or 3rd tier engineering college in India will slap together three todo list applications on Resume and wants to call himself full stack developer. Internet is fast, hardware is fast, no one cares about pile of dependencies sitting beneath them.

Add to that resume driven development where every JS wants to write libraries and become github-famous. 30-line libraries will be considered a joke and it will be shameful to brag about such things in any other ecosystem.

I was also thinking that the "blast radius" for PHP was lower, as there wasn't the same culture around creating packages.

Until composer, there wasn't much of a culture around creating packages full stop, apart from PEAR, which beginners didn't create packages for.

In 2019, 11% of all vulnerabilities listed by the National Vulnerability Database were linked to PHP; historically, about 30% of all vulnerabilities listed since 1996 in this database are linked to PHP. [1]

Also early this year there were news that git.php.net was compromised and a backdoor was introduced into PHP, but lucky enough the backdoor was catch before a production release.[2][3]

1: https://en.wikipedia.org/wiki/PHP#Security

2: https://arstechnica.com/gadgets/2021/03/hackers-backdoor-php...

3: https://flast101.github.io/php-8.1.0-dev-backdoor-rce/

I was referring to security issues regarding dependency management.

The git.php.net server was also not compromised, as far as I know, from a PHP vulnerability that was active at the time.

Also, almost all websites use Javascript but not that many use PHP. So Javascript dependencies ought to be a more lucrative target?

What's wrong with being fashionable? Most excellent devs I know have great fashion sense. Neckbeards and promo t-shirts are over.

Fashionable != fashion chasing

Npm is pretty unique in the low bar it sets for security. What is really frightening is how these cases are discovered, more or less by accident, rather than by some kind of verification process that ensures this simply can not happen before QA catches it on the way to a release.

I think pip occasionally has comparable attacks, last I heard they were mostly from typo-squatting packages though.

It seems like this was caught soon because it broke many builds. Imagine if this change was hidden better.

Correct:

[0] -> Error: Cannot find module '/Users/me/.npm/_npx/27078/lib/node_modules/@svgr/cli/node_modules/coa/compile.js'

What happened there was that he got the broken update, 2.0.3 which just referenced and used compile.js, but didn't include the file.

Then 2.0.4 came out which included compile.js and compile.bat.

Had he updated a couple of minutes later, this error would not have appeared. Not sure if /Users/ is a MacOS thing, but it is a Windows path structure, which might indicate that he was running this on Windows. And in that case he would have been compromised.

[0] https://github.com/veged/coa/issues/99

What a worthless advisory, how about sharing who could possibly be affected at the very top, or at least anywhere?

Going to the issue, it seems the `preinstall` field was changed to `start /B node compile.js & node compile.js",` which means this would only run on Windows machines, everyone else seems to be unaffected.

Here is how you can find out if you have the affected package on your machine/instance:

    find ~/projects/ -name "*coa*" | xargs -I {} jq .version {}/package.json 2>/dev/null

Assumes you have `find`, `xargs` and `jq` installed, will print all versions of coa it can find. Seems any version above 2.0.3 is bad.

Edit: is anyone sitting on the source for `compile.js` as mentioned? Would be interesting to see.

"Bleeping Computer" published screenshots of it (and also has some analysis),

https://www.bleepingcomputer.com/news/security/popular-coa-n...

This should be a top-level comment, if not a post in its own right - it explained the entirety of the situation way better than TFA.

That post doesn’t say much about `coa`, besides “new versions started appearing and builds started failing”. The bug report linked from GitHub advisory does a good job of describing the issue, though: https://github.com/veged/coa/issues/99

That's fair, but it describes the change in great detail, and makes it easier to figure out that the primary issue was only on Windows systems.

So if I’m reading this correctly, only Windows hosts are directly affected by the malware. On macOS and Linux one only needs to rollback to a healthy version of the package?

-name "*coa*" is a bit too eager. It will flag other packages too, like the fairly popular babel-plugin-nullish-coalescing

If you have an index for the locate command it's probably easier to do:

locate "/coa/package.json" | xargs -I {} jq .version {} 2>/dev/null

maybe print the command (`-t`) xargs executes, to make identification easier.

It's sort of a moot point since if you have an affected version you have to assume your whole system is compromised anyway in the case of this type of issue.

On Windows (with powershell):

    gci -r -dir | where name -eq coa | % { gc $_/package.json -EA si | ConvertFrom-Json | select version }

It seems that all of these should be cryptographically signed by a developer's private key before publication and then verified by others before use. Is that not the case?

There seems to be a call for action [0] directed at NPM.

[0] https://www.change.org/p/npm-please-secure-package-releasing

I agree that all software package repositories -- NPM, Rubygems, PyPI, Maven, NuGet, Crates, etc -- should have two minimum baseline security policies:

1. All accounts are MFA, no exceptions.

Only CI/CD usecases justify some laxity. Only push tokens should be non-MFA-able for CI/CD purposes, and they should only be usable for push. Tokens should only be obtainable with MFA.

Ideally these too would be part of a multi-step OAuth2 or OIDC refresh token to access token flow, so that any given access token is only used once and CI/CD jobs never get to see the refresh token.

And while we're on tokens: they should all expire, after a short period of time, without exception.

2. Signing packages is opt-out, not opt-in.

The two reasons package signing is relatively rare are that (1) in most ecosystems it is unwieldy, (2) not actually that effective at providing guarantees (self-signed certs are no better than a pinky promise).

But once these are solved, and I believe they can be, making signing opt-out means that we'd see signing rate percentages in the 90s, instead of languishing in single digits as they do today.

If you work on NPM and would like to swap notes, I would love to talk to you with my professional hat on -- email in profile.

and who is going to actually audit all that garbage? lmao

NPM seems to be a lot of issues https://github.com/advisories?page=1&query=malware

The Coa NPM package has 8.8 million weekly downloads. The vast majority of the downloads is from being a dependency in other packages.

Is it possible to check how many downloaded the compromised versions?

https://www.npmjs.com/package/coa

Why can’t there just be multiple curated repositories like how Linux distros do it?

Having NPM just be a free-for-all is a ticking time bomb. It is only a matter of time before an event like this results in something very serious.

There are just too many micropackages to properly look at. A basic react app created using official method (CRA) was 200MB something last time I tried.

NpmJS echo system is cancer.

Scale, basically. There are too many packages to fully curate.

Your periodic reminder that modules have way more authority than they need by default, and that there are ways to fix this: https://medium.com/agoric/pola-would-have-prevented-the-even...

(Of course this malware was in a preinstall script, which should also be disabled... but any module you import in a node app can do bad things when you run your app, preinstall script or no.)

If I'm reading this correctly, the malicious code was new (higher) versions of the releases.

Would this mean any project using a package.lock/yarn.lock was 'safe' going through deploys? So only new installs and builds without lock files could have grabbed the higher version?

If so, I wonder if it's hard or impossible to swap a release version on NPM. Seems like that would hit a much wider audience before being detected.

My read of the headline was that this was malware in embedded hardware electronics systems, or describing some exploit / attack surface for same.

May I suggest that a clearer phrasing would be ‘Malware embedded in Coa’? Or is ‘embedded malware’ a somewhat confusing term-of-art in the cybersec community?

As bad as this may sound, this is why a love Open Source, npm and the JavaScript ecosystem. It super easy to audit and check the code.

What is missing is more automated and recurrent checks in all the packages and downstream dependencies.

I just can't agree with this. The problems npm has are not new, surprising ones. They are happily letting people upload malware.

https://my.diffend.io/npm/coa/2.0.3/2.0.4/

In 2021, why on earth does such a change not trigger a review before release?

I know npm feels like wild west, but you can audit. Its quite a challenge to review imany of the C, C++, libraries out there that are just a .zip file stored in a website.

My point is that: Npm is auditable, trackable. I'm not challenging the bug itself, neither the security issue..

Trigger a review where and by who?

The receiving server would trigger it. The first review should be done by the owner, who should be contactable via email and authenticate via 2FA and acknowledge the modification.

Then I'd guess that Microsoft has enough information with NPM's history to train an AI. Specially the modifications made in these versions could easily trigger suspicious activity.

Did you look at the diffs?

Also, three years of inactivity and then a sudden upload should easily trigger a manual review, even if it is by automatically opening an issue with a review request on that project's GitHub page.

That exists already, for example github does it automatically for you.

However, that is dependent on first finding and flagging issues, which is exactly what this post is about.

This is a really odd comment. Npm, and the strange, insecure JavaScript packaging ecosystem is the reason this happened in the first place.

I don't see how this is an NPM issue at all. If the attacker made a Linux version and stuck it inside a Makefile, plenty of devs and build servers would end up executing the malicious code.

Yes, but good luck getting kernel.org to distribute that. They're not perfect (nothing is, and kernel.org has been compromised in the past, though quite long ago, and even then it did not lead to compromise of the Linux kernel sources), but to put the Linux kernel maintainers on the same level as the people maintaining the NPM distributions is a bit much.

On npm, the average application uses hundreds (or thousands?) of dependencies, none of which are audited by the application author. On Linux, ffmpeg uses dozens or hundreds of dependencies, but they're each trusted and audited by the Linux distro packaging the dependency packages, who would hopefully catch malware shipped in a Makefile. And whatever vendored libraries a C/C++ app bundles (so you can build it on distributions that don't ship the library or a new-enough version) are usually self-contained rather than having transitive dependencies. As a result, a C++ app's author count grows linearly with the number of dependencies you bring on (and you generally evaluate each author's trustworthiness, and hopefully skim over the library and build scripts as well), rather than polynomially in the dependency tree's branching factor and exponentially in its depth (you generally don't evaluate the trustworthiness of your transitive dependencies' authors, since there are so many and they're not shown to you when you add a dependency).

Additionally, an app's vendored dependencies don't automatically update until you choose to update them. And when you update a vendored dependency, you have a chance of finding malware when you review the diff before committing the update. The downside is that vendored dependencies introduce friction in updating your dependencies (I've experimented with extracting tarballs, git subtree, and git subrepo, and generally avoid submodules, and they're all somewhat awkward), and don't automatically receive patch/security updates like distribution libraries do.

I'm more familiar with crates.io and PyPI than npm, but I think they all work similarly (unvetted package repository allowing libraries to depend on other libraries, and transitive dependencies get pulled in when you install the library, and can be updated without the library author's knowledge). The primary difference IMO is the degree that packages use transitive dependencies vs. building/copying the subset of code they need (the average Python app uses dozens of dependencies, Rust apps use hundreds, Node apps use thousands).

My wish is that for each language, a team of Linux distribution maintainers or community members would "sign off on" a set of trusted packages, each of which depends on only trusted packages. And when a trusted package gets updated, the new version isn't marked as trusted until the maintainer verifies the diff isn't malicious. Then the language package manager would add a mode where apps or libraries can choose to only depend on trusted packages (which only depend on trusted packages, etc.).

(In the Rust world, where the compiler and package manager are developed by the same non-commercial organization, I hope the Rust organization will create the team handling "trusted packages"; sadly they seem opposed to it because they consider it favoritism, but IMO it's a good thing for Arch to package wget and curl, but not "joe's 1337 haxxor tool". In the Node world, V8 and node.js and npm are different groups/companies I think, so I don't know which organization is community-led and neutral enough to make a trustworthy host for this effort.)

Crev takes the "web of trust" approach, where everyone can choose who to trust. However, it doesn't ship a "deny untrusted dependencies" mode as part of Cargo. And as with PGP webs of trust, I haven't met any library developers in person, don't know which developers sign off on packages, and would prefer having semi/centralized institutions (like Linux distributions) who stake their reputation on the non-maliciousness of packages they ship, and allow different people to agree on which packages to trust. Having institutions reduces the responsibility placed on the individual app developer. Right now, you need to trust an distro to take responsibility for an OS, and a language team to take responsibility for a compiler. With crev, you'd have to also search for trustworthy people who've reviewed enough package that you can feasibly build apps using only those dependencies, instead of having the distro or language team provide that service out of the box.

Maybe I'm just too lazy to find trustworthy people. But currently there are not enough Crev reviews at all. Searching through https://web.crev.dev/rust-reviews/crates/, there's no mention of gtk and pipewire, and rusqlite and sqlite are reviewed but not libsqlite3-sys. And if I were to start writing reviews, would people trust me?

My hope is that Linux distributions will enforce a policy of only packaging applications where all dependencies are trusted packages (on top of reviewing the app's own codebase as usual). As difficult as this may sound, this is already the norm for C/C++ applications, where it's easy to find either distribution-packaged or self-contained libraries, and there are organized teams of maintainers selecting trustworthy packages, auditing and packaging libraries based on user demand, and telling audio libraries that require 70 packages to bind to PipeWire to clean up their act. Unfortunately, in languages where package managers and communities have embraced unvetted transitive dependencies, it's difficult to even find libraries with small or trusted dependency trees. This normalization of deviance has seriously harmed our ability to understand and/or trust the code making up our apps, and it will take changes in our package managers and community norms to restore trust.

This eloquently describes the root cause of the problem.

I know npm feels like wild west, but you can audit. Its quite a challenge to review many of the C, C++, libraries out there that are just a .zip file stored in a website.

My point is that: Npm is auditable, trackable.

I'm not challenging the bug itself, neither the security issue..

What makes npm more insecure than other packaging systems?

I would say that it's not that different from others I've seen, just more visible because of the size and activity of the repository.

One thing NPM does (and I believe Python too) is to allow install scripts -- this has been a reliable vector for attackers to steal credentials. Not every package repository system has that.

authors cannot revoke their compromised keys to immediately halt all distribution, and you don't have any process to verify package<->author ownership beyond the upload secrets.

Question for all folks downvoting this. How do you perform same audits with C++ libraries? CocoaPods? Pip?

Using npm is like russian roulette. Someday it makes your head hurt really bad!

What exactly is the malicious code? I assume it's in `compile.js` and only can be found in published (now removed) npm package instead of source code repo?

There's some details here:

https://www.bleepingcomputer.com/news/security/popular-coa-n...

>"Based on our analysis and information seen thus far, the malware is likely the Danabot password-stealing Trojan for Windows."

Is https://www.virustotal.com good? I saw it referenced but never used

It gives you a good glance how likely it is that something is a threat - if an overwhelming majority of AV products detect it, it's quite likely that it's bad, and if only a few products find it, it's either very new or really just a bunch of false positives. Like many things, there's no simple "yes" or "no" answer, thus interpreting Virustotal output still requires some literacy on that topic. The site can also help software vendors to find which AV products reject their software, so that they can file false-positive requests to those AV vendors.

Yes.

There's signal if there's detections

There's signal if something is old and there's no detections

There's no signal if something is recent with no detections

It's very good.

Looks like you might be compromised by this rogue Coa package if you use Windows and you installed or updated npm packages on November 4.

How do you add a new version to npm? Was the devs account hacked or how does that work?

The most common ways these things seem to happen is either password reuse with no 2fa or that the npm token (in ~/.npmrc) was harvested by another compromised package/program. IIRC there were a few that were due to phishing too.

Almost certainly one of these. It's not a typosquatting attack, since it's an existing package. And it's not a repository compromise, since they had to create new versions instead of silently altering an existing version.

It's been suggested a recent Travis CI breach was responsible. https://github.com/veged/coa/issues/99#issuecomment-96169688...

How long were the compromised versions available from npm?

Reply on news.ycombinator.com