Malicious remote code execution backdoor discovered bootstrap-sass Ruby gem | Hacker News Reader

Malicious remote code execution backdoor discovered bootstrap-sass Ruby gem

snyk.io

147 pointsqzio59 comments

I really don't know what we can do as developers to prevent this. What else could be done? Is there any package manager system that isn't vulnerable to an individual being compromised? Having an audit-tool is only useful after you've been compromised and the vulnerability is discovered. Some kind of vulnerability scanner in all pulled in code might be one idea, but much like the cat-mouse battle with AV's, I don't think would be very successful.

Ideas (shoot them down): - Somehow tie keys to location. - Require two-step verification for all commits for packages with over 1000 downloads. (Assuming their phone isn't compromised if the keys are).

For this problem, where the resulting code does not correspond to the published source code, reproducible builds is an adequate solution.

https://reproducible-builds.org/docs/buy-in/

When it comes to verifying this, we need to have independent rebuilders that build these packages reproducibly, and can attest these rebuilds in a secure manner. I have been working with NYU Secure System Labs to try provide this to debian packages and we currently have a PoC for such a system.

https://reproducible-builds.org/docs/sharing-certifications/

https://ssl.engineering.nyu.edu/blog/2019-01-18-in-toto-pari...

Not trivial to do when the build changes each time. For example adding the build number or the date of when it is built.

But I think this goes in the right direction.

Already adresse in the documentation in the form of "SOURCE_DATE_EPOCH". Adding these things to projects in general is terrible.

Sure there are ways to do it, but every piece of software needs to be aligned. What I was trying to say is that even if you have the tools you probably cannot "just use the tools".

You will likely have to create patches for those piece of softwares that create a different artifact every time are built.

There are a few other challenges but in my humble opinion, it is worthwhile to strive to write your own software with "reproducible builts" in mind so everybody can verify if something was altered.

We are well aware of the challenges. Read a bit into the documentation and the efforts that has gone into achieving reproducible builds on different distributions.

This is hard on distributions as we are suppose to support all the ecosystems. But for rubygems, pypi, go, rust, npm and so on this is a easier problem to solve if they want to. But even with all these challenges, Debian is at 94% reproducible packages, and Arch is edging closer to 80%.

Introducing non-determinism into your build process is something that should be removed, and all ecosystems should be aligned in this effort.

Weekly blog posts are written about the current efforts. https://reproducible-builds.org/blog/

Again, we are on the same page. 80% is what I was expecting. 94% is very good!

Unfortunately a few well know projects do not let you compile from source easily. Brew for example is going in that direction.

Debian had mandatory use of OpenPGP by developers for twenty years, although I don't think it's linked to strict package signing.

Of course, this adds a big deliberate barrier to participation - newcomers have to get their key signed by an existing Debian developers. There are real security/usability tradeoffs, and at the moment people seem to have default-chosen "anyone can add packages really easily", resulting in a Cambrian explosion of packages.

Keys are somewhat useless in this scenario. Developers are hacked all the time and not all of them pass phrase their keys. And if someone is going through the effort to backflip a popular package via a developer they’d probably keylig the pass phrase too.

Lots of things could be done, it's whether people are willing to accept the tradeoffs, which they haven't been (in general) up until now. Unfortunately, modern software development has heavily embedded the idea of using large numebers of un-reviewed open source libraries, changing that assumption would be very difficult.

A non-exhaustive list of things that could be done...

- Improve repository security (e.g. stronger authentication requirements). Problem here is most of the repositories are non-profit and this costs money. Also there's a risk that if you increase friction for developers, they'll go elsewhere.

- Require library signing. This has some potential benefits (may even have stopped this attack) but again increased friction for developers and management overhead for repositories.

- Curation of libraries. Actually pay some people to review the code in the libraries. This doesn't scale easily when you consider the volume that places like NPM have.

Also (on some runtimes):

- Sandbox code modules

If a library doesn't need any permissions, and it's not granted any, then attempts to e.g. access the filesystem or network will throw exceptions. The JVM can do this, but it's not well documented how to use it.

The java security manager is a cool concept but sadly not very ergonomic to use.

> This doesn't scale easily when you consider the volume that places like NPM have.

Found the blanket statement for npm.

How about instead of reviewing 1 library of 10k lines that does everything, you review 10 focused modules for a total of 1k lines?

Also modules like left-pad don’t change that often so you probably have to review only changes in the top modules.

We need to stop this mentality that dependency count should be low. Look at total LOC and you’ll probably find that it’s not as bad as people like to paint it.

A high dependency count does increase your vulnerability surface area. All it takes is one weak or reused password on an account without 2FA enabled for a malicious package to be uploaded. You're more vulnerable to this when you depend on more individual publishers to get it right.

2fa should always be on.

1) Signing should always happen. Publish time + hashes of external resources should be taken into account to prevent meddling with things after the fact.

2- a much harder solution) Don't use so many libraries.

In this way I think it's borderline impossible. Of course this vuln targeted `bootstrap-sass`.. it's basically a default gem for new rails installations since bootstrap is so commonly used.

Actually, rails 6.0 not only installs dependencies for you upon generating a new app, but it downloads yarn and installs 602 dependencies too.

It's a miracle we do anything as programmers.

The only way to completely prevent to get malicious code in your products is writing all code by yourself including the interpreter/compiler and maybe the operating system too. Machines should be off the Internet of course and security should be there 24/7. And of course you need to have a very high bar and a paranoid process to select the people you hire.

But now you will move at the pace of a bacteria on a snail. But since the company will probably introduce vulnerability bugs in the code they write, it is probably not a very good idea.

We are using open source and free packages to save time and move fast. Security and speed/usability have tradeoffs.

The paranoid will do everything in their knowledge to protect themselves but they will move slowly. The naive will not do anything and sooner than later will get burnt.

Assuming you trust your current modules and you can trust that they will not be altered, you can lock down the dependencies. But now monitor and updating a library when a bug or vulnerability is found in a modules become another burden.

Those actors that have the budget, patience and the motivation will always find a way in.

I sometimes feel quite sad about the real situation but there is not such a thing as perfect security.

That said some of the suggestions I see here will dramatically improve the situation with a relatively small effort. 2FA and reproducible builds for example.

> Is there any package manager system that isn't vulnerable to an individual being compromised?

One that comes closer is vcpkg [1], which is the Microsoft-run package manager for native C++ packages. It works by having the list of packages and recipes to build them (which it calls "ports") stored in a single repository. All requests to update them are submitted as pull requests and must be approved by Microsoft to enter the main repository.

I'm not necessarily saying whether this is a good or bad idea. Certainly it is a very different situation: only libraries that are already popular are considered, often the ports are submitted/maintained by different people than the library authors, and writing a port to build a library on multiple platforms may require significantly more work than just pointing at an existing GitHub repo. I just point to it as an alternative idea that has had some success.

That's how Homebrew[1] works as well. Any external file pulled during installation requires a matching hash in the main homebrew repo to prevent tampering after publishing.

[1] https://brew.sh/

I propose the radical idea of drastically reducing the number of transitive dependencies we rely on. I don't think that it's okay that I need hundreds of megabytes of unreviewed libraries for running relatively simple programs.

> Require two-step verification for all commits for packages with over 1000 downloads.

Just require 2fa everywhere. We're way past the point where password-only is enough if you distribute software.

Package distribution platforms could also ensure that your package comes from a tagged/signed commit in a selected repo. Reproducible builds could help here too. Finally package signing itself and possibly WoT for authors. (Where either the author has to trust the dependencies or you have to explicitly trust all of them)

This is going to practically stop remote attacks with stolen credential. But there are a few other attacks that it will not stop. For example the NPM event-stream kind of attacks.

There are some things to do, but not if you want to trust random third parties. In which case nothing will save you, because not only you have to trust them, you also have to trust they are not compromised and have impenetrable security.

So, not trusting third parties is the only way and with that language level capability based security can help. Although this is pretty far from common programming languages where any function can access any file, environment variable, global variable, constant, socket, etc.

For a start, you could detect redflags like a new "eval" call in a minor update and force manual verification...

That would be easy to workaround.

I really like the idea of a cryptographically provable signature for a set of coordinates in space and time, but not sure how such a thing is possible.

See my other comment, rust already has a decent system (unofficial)

It's worse than I suspected.

Rails 6.0, when generating a new app:

- installs 102 ruby gems (without being asked)

- installs yarn & 602 yarn dependencies (without being asked)

The modern programming industry is a joke. We call ourselves "engineers" without any certification or oaths of responsibility, and basically ignore security beyond user credentials at best.

Someone please tell me `yarn list | wc` showing 2,800 transient packages is incorrect.

[0]: https://gist.github.com/azah/c219844f95243cdfdb3b352ad3dee2f...

We definitely trust more than we should.

And it is worse that what you just found out. Did you think about all the software and dependencies that are installed just to get to the point where you can install Rails 6 (ex: Linux)?

What about your coworkers? Are you sure that you can trust all of them and nobody will not sneak anything in?

If you are too paranoid on who and what you can trust, you may end up tossing all your electronic devices and go living in a desert island.

Those attack vectors you mentioned are already known, and we essentially have to live with them. When it comes to "reflecting on trusting trust", we already know that it's basically impossible.

However, this does not mean we should by default be installing heaps of packages upon generating new packages. This is practically begging for a security exploit, particularly the JS packages.

[0]: https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...

I am not sure why the npm packages in yarn should be more exploitable than Linux.

- Linux is more ubiquitous than Node.js. If somebody exploits Linux, they get at least one order of magnitude more machines. So there is a higher motivation.

- Funny code in C is harder to spot than in Javascript. Furthermore ...

- A binary is harder to inspect than any module written in Javascript (even if minified).

- The code in the Linux default installation has at least 2 orders of magnitudes of the code that yarn installs.

I was very conservative in my estimations.

Considering these kind of attacks, Linux seems more likely to be already exploited by a few organizations in multiple ways.

We have been trusting too much. As more code gets written and many more people come to learn to program, the number of supply chain attacks will increase. And at the same time, as the security know-how is easier to be accessed on the Internet, we will get smarter exploiters capable of hiding their wrong doing.

`bundle install` outputs `Bundle complete! 17 Gemfile dependencies, 78 gems now installed.`

Just doing some eyeballing:

- These look like the 62 gems in the production group: https://gist.github.com/gkemmey/6d6fcd381596b1b355d16e1e3120...

- Of those 62, how many are first party (by rails or other trusted vendors)? Looks like the vast majority...

- These look like the other 16 (dev / test groups): https://gist.github.com/gkemmey/6d6fcd381596b1b355d16e1e3120...

- On the JavaScript side, rails adds 5 top-level dependencies. 4 of those 5 have one external dependency between them: https://gist.github.com/gkemmey/6d6fcd381596b1b355d16e1e3120...

Of course, the fifth is `@rails/webpacker` which has `webpack`, `babel`, and friends as dependencies. And the node ecosystem is what it is, but I think `yarn list | wc` is going to double count shared dependencies, so 2800 is high.

TL;DR - I think new rails is about as minimal a footprint as possible...

Relevant links:

- https://github.com/twbs/bootstrap-sass/issues/1195

- https://github.com/rubygems/rubygems.org/issues/1941

Do people in mission critical systems (like life threatening) write code from scratch all the time without using any external dependencies? I'm sure they don't reinvent encryption.

I'd like to know how they audit their system and the threshold of saying it is adequately secure.

Rust has an effort going on (by a single unaffiliated person) to have a trust chain sort of thing per version. Cargo-cred.

I hope others copy this. This is truly scary.

For those who haven't yet come across it, bundler-audit is a very useful tool to pick up any old and/or insecure gems in your application. It should be part of your CI pipeline, or at a minimum, run locally every so often.

https://github.com/rubysec/bundler-audit

Note that it won't yet detect this specific vulnerability: https://github.com/rubysec/ruby-advisory-db/pull/386

That's great.

Is there also a bundler-meta-audit the somewhere, so I can audit bundler-audit and it's dependencies? If not, maybe we should start a kickstarter for it?

Another one in the line of inevitable supply chain attacks on software libraries.

This isn't going to go away, it's just a question of if/when the damage caused will get bad enough that people start to accept some of the tradeoffs that would be required to improve matters...

Can you elaborate on what do you think would be a good solution to this problem? In my opinion, having big central sources of libraries makes it easier for security scanning services to operate. Companies hosting libraries locally (e.g. with Sinopia, Nexus, Artifactory) have to proactively monitor them, but most commonly will let them frozen at a specific version and miss all security patches.

Really the only "solution" is to ensure that you have some way to establish trust in all the software libraries you use.

For commercially provided libraries that could be a contract specifying security requirements, with some specific measures of how to do that.

For Open source the only real option I can see is to curate your own package repositories and get a level of review that you're comfortable with. Definitely fixing versions and insisting on review before upgrade would help.

The problem with automated scanning is that they can't really find backdoors or just generally insecure code, they can find known vulnerabilities. They could use static analysis to find insecure code, but my experience of SAST tooling is that it takes a lot of manual effort to tune, it's not a pure automated scan option.

That's not to say that automated scanning provides nothing, but that it has limitations.

> curate your own package repositories [and review them]

This is a non-starter. This is even worse than "just run your own mail server".

sure but what's the alternative? Unlike mail servers where there's tonnes of good commercial solutions you can use, I'm not aware of many good options for curated software library repo's

There's tools you can use to add automated scanning to your repo's and pin versions of packages, but that's doesn't really feel like curation to me...

companies like Sourceclear seemed to me to be going down this line but that doesn't seem to be the drive of their offerings any moe.

In the end it all comes down to reviewing code you have to trust. It's just that it's too much for most code, so there has to be something that makes potentially insecure code explicit and dealt with on use and the rest of the code usable without having to trust it.

Reviewing code mitigates a lot the risk but it does not solve everything (ex: vulnerabilities hidden in plain sight).

It also implies that you are compiling everything from source code or you trust the distribution channel of the binaries.

And you do it every time there is an update that you want to use. And you have a system that promptly alert you of vulnerabilities in libraries you use.

Unless you have a handful of modules that are already self contained, it is not practical for most teams.

This is very timely: I was trying to start up an old Rails app today and when I ran the bundle install command, I got a warning that there were no sources for this gem... now I'm glad that this failed to install!

I hope this will help rubygems adopting reproducible builds. Currently their issue with SOURCE_DATE_EPOCH is not yet supported properly.

https://github.com/rubygems/rubygems/issues/2290

https://reproducible-builds.org/docs/source-date-epoch/

Great writeup by Liran, he's also a member of the Node.js security team and I recommend checking his talks about package ecosystem security.

Ultimately, I hope we end up getting (in some future) some form of "cross language" global network of versions & vulnerabilities signaling, with e.g. alerts when you install something, or at runtime, and push notifications for all "subscribers" (outside of a paid product).

OWASP [1] maintains npm, maven (and nuget, I believe) plugins for detecting use of packages with known vulnerabilities, and a vulnerability db, of course.

[1]: https://www.owasp.org/

Talk here: https://vimeo.com/274629722

> We assume that the attacker has obtained the credentials to publish the malicious RubyGems package from one of the two maintainers, but this has not been officially confirmed.

Didn't this _also_ require a commit to the relevant Git repo? Or is it possible to upload a tarball directly to Rubygems without it being backed by a git repo?

5th Paragraph in:

> The backdoor was wisely hidden in the 3.2.0.3 version that was only published to RubyGems and no source of the malicious version existed on the GitHub repository and allowed remote attackers to dynamically execute code on servers hosting the vulnerable versions.

Relevant I'm working on a system for auditing open source libraries: https://github.com/dpc/crev . Currently focused on Rust/crates.io.

Anyone maintaining a library with a large audience of consumers should be mandated to use 2FA/MFA, I can only assume from the information here that they only had a set of credentials for authing :-(

Having every package management system requiring 2FA to publish is a great idea. It may be a problem though when the build happens on a CI system.

This is scary. Gems don't have any audits?

Not this kind of audits. Is there any major language for which all third party modules are audited for security before they are released? I would be surprised.

gems don't, NPM doesn't, PyPI doesn't, NuGet doesn't.... you get the picture.

Basically all apps using package repo's (i.e. all of them) are relying on massive piles of unaudited 3rd party code with usually no idea of provenance.

Reply on news.ycombinator.com