NPM security update: Attack campaign using stolen OAuth tokens

All of them to at least some extent and several we will soon have names for because they are still emerging.

I agree fully with you but I don't think secret management is as easy/cheap as some people pretend. On AWS, for example, each secret you store is $0.40 + $0.05 per 10,000 API calls and that can add up if you only have 1 api key/password/etc per "Secret" (for an individual at least, I hate bleeding off $10+/mo to store tiny bits of strings). Then, once you have the secret stored, you need setup roles/policies to be able to retrieve them.

I have this setup pretty well in my code now but getting there wasn't simple or easy from my perspective and keeping the list of secrets your IAM user can access up to date can be a pain as well.

I'm working in a lambda environment so my options might be more limited but I'm interested to see how other people are solving this issue (maybe specifically for small/side projects). As it stands my lambdas all get a role applied to them that gives them access to the secrets but something not AWS-specific would need a "bootstrap secret" to be injected before the code could call out to the third-party to get the other secrets. For Lambdas I suppose I could inject that "bootstrap secret" in via environmental variables but now I've got a new issue to deal with. Injecting at build time via something like GitHub Actions Secrets is an option I guess.

All that to say, while I agree secrets should never be in source, in practice it's not super easy (I'd love to be proven wrong, maybe I'm not doing it right).

A rule of thumb is: if the key is pushed into a git remote you should consider it compromised and roll a new key.

This is also a thing that will not get past some interviewers.

That is one thing that RFC8959 is intended to solve. If you see "secret-token:" in any logs after running tests you flag that as a problem and fail the test.

I agree it should be this simple, but I'd bet they have tests like this and unfortunately it's never quite this simple in a production system.

I've always wanted to apply strong type systems to this problem – wrapping sensitive data in types that do not have the ability to be printed to logs would theoretically allow you to know after type-checking that passwords can't be output to logs. However again I think this is wishful thinking as a password needs to be sent somewhere at some point, and that creates places where issues can occur.

I've worked on a service that handled credentials where we added tests like this to try to catch if a log statement gets added containing the username/password. We used a few end to end tests rather than attempting to include something like this is the unit tests for every function.

Our tests would set up the app's full context, get a hook into the logging framework to watch for log statements, then make requests to the service containing a set of dummy credentials, like { username: "foo", password: "bar" }. If a log statement containing "foo" or "bar" was detected the test failed.

It's not going to catch every type of issue, but at least some potential footguns can be preventing this way.

This happens all the time in a lot of companies, you just don’t hear about it.

If this was a clever hack, I'd agree with you, shit happens.

But for gods sake, having secrets hardcoded in VCS??

You seem to understand threat modelling. What's the threat towards one of the biggest and most used package registry?

It's not like npm Inc just started running the registry. They have been doing this for years. To let such a beginner mistake risk the supply chain of basically the entire JS ecosystem is not only sloppy, it's completely unprofessional.

What can we do in the short-term? I'm not sure, but I hope smarter people than me comes up with some solutions ASAP before a compromise like this starts actually impact developers using npm.

Serious question: what’s the difference? Do Linux distros actually audit packages very much? Supply chain attacks in Linux distros are pretty scary, as you can basically expect them to run install scripts as root.

A fraction and based on my old Ruby gems memories, fairly out of date.

I remember that somebody posted a reply to a comment of mine here on HN years ago saying that s/he was rebuilding every single gem as deb package before deploying it in production and that was the only sensible way to do. I don't think it adds much to security unless they also read all the code, but it's a lot of work that none of my customer is going to pay me for. I also probably don't want to start a profession of deb builder for Ruby gems.

You cannot download React from apt/yum AFAIK.

What do you expect the community to do?

Stop using thousands of packages? Start vetting packages, as if security was important?

There are already dozens of us saying it is possible to not have too many dependencies, and vet packages before installing. But every time we open our mouths we are treated as if we just escaped some sort of insane asylum.

At a place I worked in the past we used to have a 40-line microservice using plain-node without any dependencies. That was by design. One junior dev took it upon himself, in their spare time, to convert the whole thing to use some js MVC framework, complete with a full-blown build process, transpilers, and all the nine yards. There was a big discussion in the PR and a lot of juniors complained that we should migrate because they "didn't learn plain node.js in college".

We can't have nice things anymore.

For once, I don't think this highlights an issue exclusively specific to JS.. this could've happened to any package system that Github owned when the attacker was able to pivot after accessing the private repos.

  > Will this be met with a shrug from the JS community? 

Go read the comment that begins "Top 10 maintainer here". Not even a shrug.

Was this incident facilitated by something inherent in the JS ecosystem? I have the impression it wasn't.

The JS ecosystem sucks, but anyway, not particularly their fault in this case.

I’m of a bit of an opposite mind on the many, and usually very public, NPM security issues. Because from my experience the JS ecosystem, and it’s woes, teach a lot of people to never trust the part of their operation that is coming from someone else. I mean not everyone, obviously, but in my anecdotal experience it’s far more common to see good package control and review processes for JS than any other language, well except for maybe Python when the Python is done by software engineers and not “data-scientists”.

Supply chain security is immensely important, and I encourage you not to learn about it the hard way like I did. Which somewhat ironically happened in the .Net ecosystem when one of our trusted Nuget packages got hacked many years ago. Now, I could be mistaken and I hope I am, but I suspect that if you ask a Java, a JS and a C# developer if they trust their ecosystem, then only one of them is likely to say yes.

So no, there won’t be some great revelation in the JS community. The best you can hope for with stories like these is that fewer developers feel like imposters when they realise that GitHub stores plaintext security assets in their logs.

As a member of the JS community: shrug. I revoked my OAuth apps on Github, changed my passwords. The tarballs are unaffected. Not worried.

What would you expect the JS community do after this? What would you do?

Yes, because this fundamentally wasn't an attack against NPM or any specific package manager. This stemmed from a breach at Heroku.

I take it you’ve never been involved in a breach investigation. Figuring out what an attacker had access to and whether they exploited that access isn’t trivial, especially for a heavily used service like npm. To say they were “sitting on” information while probably tens if not hundreds of engineers assisted in making sure the investigation was complete and accurate is uncharitable.

TFA starts with "On April 15, we published a blog[0] detailing an attack campaign ..."

So, there was a blog post on April 15, at least.

[0] https://github.blog/2022-04-15-security-alert-stolen-oauth-u...

On one hand it's good to have an organization with lots of engineering resources behind these, but I have to wonder whether that much stuff in one place (Microsoft in this case) won't bite us down the line.

If you do client side hashing then the hash becomes the password, it only helps with password reuse on other services. From your service perspective the security issue doesn't change much.

> > "On April 12, GitHub Security began an investigation that uncovered evidence that an attacker abused stolen OAuth user tokens issued to two third-party OAuth integrators..."

> Too vague to be useful. Stolen from whom?

Heroku and Travis CI. That information is literally in the end of the sentence you cut off:

"On April 15, we published a blog detailing an attack campaign utilizing stolen OAuth user tokens issued to two third-party GitHub.com integrators, Heroku and Travis CI."

> The password hashes in this archived data were generated using PBKDF2 or salted SHA1 algorithms previously used by the npm registry. These weak hashing algorithms have not been used to store npm user passwords since the npm registry began using bcrypt in 2017.

Which is so frustrating. When you upgrade your hashing algorithm, always always _always_ immediately remediate the weak hash mess by hashing your weak hashes with your new stronger hash, and turn the login check into

    bcrypt(sha1(user-entered-password)) == stored-bcrypted-sha1

you can then upgrade them to a straight bcrypt if the check succeeds, but keeping the weak hashes on disk indefinitely until the user logs in (if ever!) is such a risk.

From https://github.blog/2022-05-26-npm-security-update-oauth-tok...

“The password hashes in this archived data were generated using PBKDF2 or salted SHA1 algorithms previously used by the npm registry. These weak hashing algorithms have not been used to store npm user passwords since the npm registry began using bcrypt in 2017. ”

What does it mean for a company to "know better" than to do something?

It means some combination of the following: people don't make the mistake in the first place, the mistake is caught in code review, the mistake is caught in audits, the mistake is caught by automated tooling...

I don’t understand. Just about everyone rolls their own auth.

This is awful advice. I’d never trust my authentication needs to external providers who can hold my user information hostage.

Rolling your own has some benefits, though. Such as not being as big a target as a provider that services many sites, and being more in control of the auth data, and not being reliant on a service that's beyond your control to not suddenly stop working or shut down.