Unauthorized gem takeover for some gems
github.com
github.com
- You can’t trust the package registry because of security flaws in the registry itself (as seen here as well as in NPM just a few months ago[1]). The CDN can be hacked, or an insider can attack the infrastructure.
- You can’t rely on code signing since a maintainer can go rogue and sabotage a package at any time (as happened with colors.js and faker NPM packages in January[2]) or a new maintainer could be added to the project and possess a valid signing key but turn out to be a bad actor (as happened with the event-stream NPM package in 2018[3]).
- You can’t audit every line of code in every dependency because the cost in terms of time and expertise is prohibitive to all but the biggest organizations (e.g. Google) and the most security sensitive applications (e.g. certain crypto projects and financial applications).
The two solutions I’m most excited about are (1) auditing package behavior with static analysis to detect when package behavior changes (e.g. new network connections, filesystem accesses, install scripts) like we do at https://socket.dev (disclosure: I am the founder) or (2) package sandboxing of which Lavamoat is the best example (however the performance impact is too high to apply this to every package in an app at present, and it also requires maintaining a policy configuration for each package).
[1]: https://www.theregister.com/2021/11/16/github_npm_flaw/
[2]: https://www.theregister.com/2022/01/10/npm_fakerjs_colorsjs/
[3]: https://www.theregister.com/2018/11/26/npm_repo_bitcoin_stea...
I think you can probably split the two areas of interest into:
1. A package maintainer's credentials are compromised
2. A package repository is compromised
And the two attack vectors into:
1. The build script(s)
2. The runtime library
You can cut off the "repository is compromised" path with signing. An attacker doesn't have the maintainer's private key so even if they can modify source/packages on the server they can't "trick" the client into verifying it.
(1) is harder. Let's assume we have package signing, we know that any package we have was signed with a key that we hope only the maintainer has access to. At this point, either the maintainer is compromised or malicious. We can make compromise harder in a few ways, but should ultimately assume compromise.
One way to reduce compromise is to have the signing key stored on a hardware token that requires proof-of-presence for signatures.
Still, at this point we have "build script bad" and "library bad". Both are much harder problems, with solutions that you've alluded to to some extent - that is, sandboxing behaviors.
What this requires is a way to say "this code can do these things". This is how browser extensions / mobile apps work - they have to declare their permissions and you have to ack them any time they change.
Doing this for build scripts isn't too hard. You can run the scripts in a "hermetic" build system and have each script execute serially in a restricted environment - if one needs networking, give it networking, etc. There's no native support for this but imo it wouldn't be that hard to add it in.
Doing this for libraries is much harder. You'd need a native capabilities system in the language, and changes to capabilities would break the API. But sandboxing entire processes isn't that hard. The vast majority of services don't require egress to the public internet, meaning that an attacker is already going to have a hell of a time if they just get a shell into some box that they can't even communicate with. So I'd say start there - limit processes to what they can do and that limits the impact of a compromised library.
So altogether, none of these approaches seem super hard. We have signing, sandboxed builds (they can be pretty loosely sandboxed tbh - do a 'fetch', and then cut off internet for build scripts/ limit fs access), sandboxed services.
It'd be nice to have something more robust, but today you can do everything above without a ton of effort.
Supply chain vulnerabilities are becoming a greater problem by the day. I think Solarwinds demonstrated this greatly.
Reflections on Trusting Trust: https://dl.acm.org/doi/pdf/10.1145/358198.358210
Previous discussion: https://news.ycombinator.com/item?id=26887670
Like the founder of socket.dev, I think that static analysis will play a role. However, I don't agree that that's the entire answer.
The static analysis has to be in the programming language used in the build. The language must have fine-grained permissions built into it, and those permissions should be able to be used both at compile time and runtime.
The build system should take advantage of the programming language's fine-grained permissions to allow the user building the software to restrict what the software can do at build time. It should be possible to decide, for example, that the build can only access files in a particular directory, and if that is violated, the build system will error. Same thing with network access.
The package manager should take advantage of the build system to build all software from source as far as possible, and to ensure that the package uses the build system and programming language to ensure the package cannot stomp on the user's computer at build time.
That is the minimum before these issues have any hope of being resolved.
But I still think that is not enough.
Say you have a dependency that you do not trust. If you could compile it into a form that could be interpreted by the same interpreter that the programming language mentioned above uses to enforce permissions at runtime, and if that interpreter had a way to transfer data to and from the code it was running, then you could have a system where you embed the interpreter in your code and use it to run your untrusted dependency. Then the untrusted dependency can be properly sandboxed.
Yes, you'll pay in performance, but you won't pay whatever you might pay if a dependency goes rogue.
That is the only way to "fix" these supply-chain issues until everyone actually vets their dependencies.
Disclaimer: yes, I'm working on a language, build system, and package manager with the above features.
(How) does that work for dynamic file access? I.e. the files are often not known at build time.
I don't know if my current design is good, so it is subject to change.
The current design is to allow the user to give the interpreter a predicate (function that returns a Boolean) to run whenever something asks for access to a file. It would return true for allowing access, false otherwise, and its arguments would include what package (module, library, etc.) is asking for access, what kind of access, and exactly what file it's being asked for. The predicate can do all sorts of wild and wonderful things to figure out if the package should be allowed access. It could even pop up a dialog box for the user in the cases it can't figure out, which would also have the benefit of alerting the user that there's a package that might have gone rogue. In fact, in those cases where it denies access, it can still alert the user, who can then do something about it.
I like this design (so far), and I think it's best because if you're going to enforce runtime permissions, you need runtime abilities to make judgments. It can't be a simple static list of files, as you rightly figured out, although it could use such static lists for the easy cases.
How would folks do that with strong enough guarantees? Surely they can't review every gem change by hand?
Given the nature of the vulnerability, they only have to audit yanks of gems with a dash in the name that were either not updated in the 100 days prior to the yank or created 30 or fewer days before the yank, which is a much smaller universe than “all gem changes”.
An update triggers the whole process again.
Yes, it is a pain to deal with, and the only way to make sure 3rd party dependencies are actually valid, and also keep working no matter what happens to upstream.
E.g. package A depends on B, if you install A, B is baked into it at the install step. That way, your active dependancies are only that which you purposely install, and some compromised package X dependancies down the chain won’t bite you when you do an automatic upgrade.
Vendoring all gems, manually verifying diffs on upgrade, and manually verifying signatures where possible is still the best practice here.
Proper firewall, TLS, strong crypto, secrets management, 2FA and regular audits. Zero trust stuff
Zero-trust requires certain scale and is expensive; the most successful high-stakes zero-trust system I know is Bitcoin.
They can still be badly behaved, but you are least control the blast radius.
There was no actual takeover of any gem. A vulnerability was found that made it possible, but after audit no such case was found.
2021 - THE year of secure supply chain.
2022 - THE YEAR OF SECURE SUPPLY CHAIN.
Working in this general space, I can say that this is only going to get worse before it gets better. Many people are rallying behind pushing towards best security practices however.
The dash requirement is interesting given it's somewhat of a pseudo prefix/namespace. Good that this was caught now and looks to not be exploited.
Events like these don’t surprise anyone who’s been following this space. It’s been a rough ride that will get rougher. Consider:
1. NPM just recently started enforcing 2FA for some packages. The NPM has yet to support mechanisms like package signing. Post install scripts are still a thing.
2. Rubygems does not support package signing.
3. Go dependency management is a hot mess.
4. Clojure dependency management is still a hot mess.
Initiatives like Google’s SLSA will help here, but there’s a whole ecosystem of stuff across multiple technical ecosystems they need to mature, and fast.
The problem with go packages isn't the security of the package manager, but rather the fact that I need to download some arbitrary go code from github to make uuids. "Pull this random package" has become the new "copy paste from stackoverflow without reading the snippet".
Rubygems has supported package signing (`gem help cert`) since very early on, and it has an install flag `--trust-policy` which can be used to verify various things, including certs (https://github.com/rubygems/rubygems/blob/96e5cff3df491c4d94...).
The experience in using it, however, sucks on every level. No one can really use the `High Security` policy level, because most gems aren’t signed. Most gems aren’t signed because there’s no clear benefit and it’s non-trivial to have shared certificates that can be used by multiple people authorized to release a particular gem. Most gems aren’t signed because there’s nowhere that public gem certs are published (there used to be with rubyforge), and you have to track down each cert you want to verify and download it separately.
I used to sign my gems, but then stopped.
Shopify has proposed a new RFC for signing gems based on sigstore. This RFC has many of the same points that I have already made as a reason for changing mechanisms. https://github.com/Shopify/rfcs/blob/new-signing-mechanism/t...
I’ve just discovered this, so I haven’t really evaluated it, but I would prefer to sign the gems I publish.
The pattern plays out with every new wave in our industry. You can even see this with the k8s ecosystem where there is a ton of complexity and opportunity for attackers. It's only later that security is taken more seriously when it could be designed in from the start. We love to mitigate things that we could eliminate in original designs.
Yes, but actually no.
"To increase confidence in the npm public registry, we add our PGP signature to package metadata and publicize our public PGP key on Keybase."[0]
Unfortunately (and perhaps unsurprisingly), this means NPM is still a single point of failure, and this doesn't add any defence in depth. I guess what you want is signatures by the package maintainers themselves, but key management is hard for both signers and verifiers.
[0] https://docs.npmjs.com/about-pgp-signatures-for-packages-in-...
What this is not is support for package maintainers or CI/CD systems to sign artifacts they upload to NPM. Supporting this would give consumers of said packages the ability to detect if a package maintainer was compromised—the signature on a new version would either be invalid or different.
> Post install scripts are still a thing
I cannot find a situation where post-install script actually pose a security issue. I find that you'll always end up running the code you just installed on your machine (either through running unit tests, or running the actual code, or whatever). Hence, the only scenario where post-install scripts are a problem is:
- You install deps on a machine that has access to security sensitive resources (think: your ssh keys)
- BUT you do not run the resulting code on that machine.
Are there really any use-case that satisfies those requirement where post-install scripts would actually result in a security issue?
In general, anyone who gathers dependencies ahead of time before deployment, so they can just copy them over. So most people deploying to multiple machines/containers would be at a greater risk from a post install script
In order to get the code in scripts executed at a predictable time – you add it there and done. In order to get the code in the library executed on the build box, you have to add it somewhere that's actually used by the environment you want to attack. Scripts allow more coverage more easily.
Plus, it's not unheard for people to run "npm" as root to run some scripts as root.
https://github.com/ossf (Lots of WG and efforts such as package analysis, scorecards, etc)
https://deps.dev/ (Implements OSSF scorecard)
So far folks have turned up from RubyGems, PyPI, NPM, Maven Central, Gradle, Drupal and I'm probably forgotten someone.
Sorry, but that just isn't true. Solutions have existed for years [1], but enterprises just weren't interested.
[1] For example Sonatype who I know about from a former colleague (ex. Sonatype employee). They have existed for 14 years.
It’s also worth pointing out that my original comment was probably less specific than it should have been—SCA isn’t new, but tools to help and mitigate risks around the digital supply chain have up until recently been fairly primitive as far as I’ve personally seen.
Many vendors can tell me a package has a vuln. Very few can tell me which vulns can actually be exploited in the context of applications I secure. Fewer still can give me a cohesive view over time of what packages or maintainers start doing things out of the ordinary.
Nexus Firewall automatically blocks malware and malicious typosquats, hijacked packages and dependency confusion attacks with algorithms now being expanded to cover self-sabotages (In fact, I first broke news on dependency confusion along with researcher Alex Birsan on the company blog and BleepingComputer). As such, before the attacks even picked up steam, Sonatype already had a solution for it and been blocking these for months - but coordinated disclosure agreement for PoC research delayed our public disclosure.
Nexus IQ/Lifecycle is more for SBOM/vulnerabilities, including those without a CVE - e.g. reported via GitHub Issues and other sources. The vulnerability scanning looks for the exact occurrence of vulnerable code rather than just flagging any and all artifacts for a given component, which makes it quite precise imo.
For SCA, there's Sonatype Lift that connects to your GitHub repo for free so you can test drive it before moving on to other offerings.
Thanks, and I hope it helps.
What are your thoughts on risks in this space? As a member of an org that thinks about these problems a lot, I’d love to hear about any novel attacks or mitigations you and your team have eyes on.
Whereas, previously only typosquatting or malware published to OSS repos might have been the primary concern, the recurring incidents today have diversified in both their type and quantity.
We now have to dedicate more time and resources to analyzing malicious packages that we'd have otherwise spent on hunting for zero-days or vulnerability research activities. These packages keep multiplying and it's become a whack-a-mole situation: every other day we report malware to the OSS repos, these get taken down, and the threat actor repeats the attack with slight variations a few days later. Additionally, copycat attacks follow, further increasing the number of malware incidents.
For example, other than typosquatting attacks, between last year and now we saw attackers hijacking legitimate libraries (ua-parser-js, coa, rc) or publishing tens of thousands of dependency confusion packages (A dependency confusion attempt against VMware VSphere SDK devs was just caught by us, along with 1000+ packages targeting Azure developers caught between March & April - our blog posts will explain it all. We've thus far flagged well over 65,000 suspicious packages including malware, dependency confusion attacks, typosquats, PoC tests, etc.).
In 2022, we are met with self-sabotage and protestware incidents that are on the rise: colors/faker, node-ipc, event-source-polyfill, styled-components, es5-ext, ... These have further complicated matters, and pushed us to fine-tune our algorithms. We can now no longer trust the original developer of a library either, as they are free to change their mind on a whim (they always were).
As pioneers of a proactive solution behind dependency confusion attacks and a company that's been consistently leading OSS malware discoveries every week, I'm obviously biased but I'll say whatever solution you implement, make sure to have something in place to protect your dependencies, components, and supply chain against these novel attacks. Vulnerabilities like Log4Shell or Spring4Shell, as serious as they are, are just the tip of the iceberg when you look at the entire OSS threat landscape that's evolving.
You raise a very good point about self-sabotage and protestware. I'm reminded of the left-pad debacle from some years back that fell in the former category.
Protest via libraries is new and interesting, but I suspect boils down to a matter of acceptable licensing, liability, and guarantees.
While well intentioned, I have some anxiety that protests through dependencies that operate in production environments could cool adoption of open source projects. The solve for this will likely be more attention paid to governance of open source components--which is something I already encourage.
What trends are you folks seeing with protestware? There is a real need to spread these messages, but I don't see many consumers of said projects being receptive to their platforms assisting without consent.
1. colors/faker followed the Log4j debacle and was more about corporations using open source heavily but not giving back enough to support the developers so the dev threw in the towel. Applications using 'colors' began freezing (entered a DoS condition) due to an infinite loop introduced by the developer in the code.
2. But with node-ipc, the self-sabotage turned destructive with the package actively deleting files on detecting a Russian/Belarusian host IP
3. event-source-polyfill, styled-components, etc. have adopted more a more "peaceful protest" approach by expressing the maintainer's views condemning the Russian war, but without engaging in outright destructive activity.
Thus far the trends have been about open source and the ongoing war.
But developers have discovered a new avenue of their creative expression (open source) which no longer limits them to simply coding the intended application functionality. And so, the questions that arise are, what will the next protest be about and if we are prepared for it?
But as you say, basically nobody uses it.
Surely not one stored in rubygems.org, since an attacker who compromises an account could just upload a new key and sign a malicious release with that.
Besides, if rubygems can write a bug that allows someone to publish to a package namespace they don’t own, it’s perfectly realistic that they could also write a bug allowing an attacker to upload a key to an account they don’t own.
SSH is a good example where the host key is trust on first use. It would be nice to have this option for supply chain dependencies too (npm, ruby, etc). So your local client would reject any updates to a package if it wasn’t signed by the original key.
Like SSH, if the key had to be rotated due to compromise, you’d have to reset that trust. But at least that would be a manual process and changes would be picked up as an error by build servers.
Agreed, I see people dismiss valuable but imperfect security too often.
Google uses a similar approach for Android packages. In order to publish to the Play Store you need to upload signed with the same key. If you can't, then you submit a support request and they run some extra checks before they allow a new key.
On challenge I don't see others mention I that an attacker could publish a small, non-malicious release first. Then any manual diff review wouldn't see an issue. The next release using the attackers key would be trusted, so manual review would be skipped, but that one would contain malware.
This requires two releases by the attacker, over a period of time. Slowing down the attack is a benefit on its own. It gives package owners time to wonder, wait who pushed that last release?
Consider that most nontrivial projects on rubygems have multiple maintainers that can publish a version. Normal collaboration models would imply that they each have a personal signing key; sharing a single signing key per project isn’t realistic (as you mentioned, rotation is another reason for this). And TOFU doesn’t work when there are multiple possible keys, such a system requires an external trust chain.
Assume for the sake of argument the above is solved. What exactly do you do when tooling alerts you that an upgraded dependency has changed keys since the last version? Either you blindly accept the new key or you investigate. If the latter hopefully you have a way to directly contact the author to verify that the rotation was legitimate. Since you probably don’t you should just compare the diff of the published artifacts. But you should have been doing this anyway, so what has the signature bought you here except false security?
Imagine a system like npm, where hundreds of transitive dependencies are upgraded all the time. Would you really want a key change in a 3rd-level dep you didn’t even know you were using to block you from installing a security fix to a primary dependency until you could somehow vet the new key? Such a system would change very quickly to Trust On First Use But Also Trust On Any Change…
I’m all for pragmatic solutions that measurably improve supply chain security. I just think changes like https://github.com/rubygems/rubygems.org/pull/2499 and https://github.com/rubygems/rubygems.org/pull/2242 which recognize the social aspect of the problem qualify to a much greater extent than thinking of crypto as magic dust that can be sprinkled on a system to increase its security.
Generally I wouldn’t be too happy using a dependency where there are lots of users with access to create official releases.
Re npm and transitive dependencies, this isn’t a code signing issue as much as it is a version pinning issue. Specifying latest as the version is a poor approach imo. Also the primary dependency should not be releasing a critical security fix as a breaking change. If it has to be a breaking change then there’s marginal additional effort to having to deal with the transitive dependency too given you’re already having to change things to support the primary dependency.
I’m not saying TOFU signatures would be a silver bullet. They, in conjunction with other security measures, best practices, etc, would help a lot though.
> Generally I wouldn’t be too happy using a dependency where there are lots of users with access to create official releases.
I’m curious, do you use any open source software? Most larger projects have a number of maintainers with release rights.
I want to clarify that I’m not against package signing. For example, I think this proposal is well-thought-out and would be a positive change. The key reason this is better IMO is that it introduces a centralized trust mechanism and a certificate transparency log: https://github.com/rubygems/rfcs/pull/37