Backdoor in event-stream library dependency
github.com
github.com
Depends on the project, I guess. Open source can always be reanimated.
> he emailed me and said he wanted to maintain the module, so I gave it to him. I don't get any thing from maintaining this module, and I don't even use it anymore, and havn't for years.
That's just plain irresponsible. He must have known how popular the library was. Taking ownership of browser extensions and common software libraries is becoming a very popular attack vector. Owners of these need to be a little more diligent when transferring ownership.
Perhaps there should be some sort of escrow jail for new maintainers where all changes are reviewed. Certainly better vetting needs to take place.
In the absence of accepted responsibility from (monied) consumers, the way responsibility is ultimately taken on has to be either to affirmatively take it on (as public service to the community) or to pay for it. Which is a sticky problem because it's hard to pay for it and it's invariably not presented effectively in terms of building a business case. Existing solutions, not to put too fine a point on it, suck at building that case. OpenCollective, the remnants of Gittip/Gratipay, etc. - incentives don't align to put money where it needs to go, and J. Random Consumer often suffers the most for it.
I have a pretty strong idea about how to solve it for-reals. At the moment, though, it's a time/money problem for me; I'm not in a place to chase the kind of (relatively small) funding that the problem probably requires. If anyone else is interested in this problem space, please feel free to email me directly; I'd be happy to chat with you and I'm very interested in seeing this done well.
I hate to have to do this, but the requirement runs right to the core of how this development model functions whatsoever:
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
I'd suggest one problem here is a user interface / packaging model issue: end users / downstreams may override the version choices made by their upstream dependencies. In the case here, the reputability of that upstream varied over time. Permitting version locking allows a chain of reputability to be exist, allowing a limited amount of trustworthiness to be imparted by upstream's selection of dependencies ("I trust this guy's package, so I trust all their dependencies too")The alternative is to use distributions that do vetting and staging.
Monitor the network, make sure you know where the data is going. Things like that.
I mean, personally, the alternative I like is a sort of hybrid model, where you pay someone else to do the vetting, like RHEL, but nobody I support is willing to stay within the RHEL package world, or to otherwise pay to really vet the packages they use, so it's defaults all the way down, and it usually works just fine! until it doesn't.
If maintaining modules earned money, that would be much more incentive to "maintain" thousands of random things you never look at, and to hand over control while keeping it in your name (which he's also being blamed for).
That wouldn't really solve the problem. Attackers would just have to wait a bit longer before they push malicious code.
Paying for a better, more consistent lunch makes sense at some point, yes?
People are motivated to work, ingredients are vetted, better prepared, etc...
Not blaming anyone, just pointing out an obvious problem by analogy.
Let it be known that it's discontinued and let the new maintainer trade on his own reputation.
I'm sure he would do it for $150k/year. He won't do it for $150k/year? Increase the price until he says "Yeah, sure I will maintain it". Or maybe he will find someone else would would maintain it for that money.
Vetting new maintainers sounds like hard work that you might not feel like doing for free no problem just don't do it and don't give them the damn name.
I can prove any of a number of identities scattered around the internet most of which are in fact my real name. Pretending that this is trust less is just not real.
Example I know from a wide variety of sources that certain projects are trustworthy even if I can only verify a pseudonym that itself is trustworthy and infer that authors other projects are trustworthy.
Accounts emails and domains are all useful tools even if not perfect.
People don't normally put years into developing trust in order to distribute malware. It's normally a low effort affair.
Not giving maintainership to random people who send you an email or selling projects to skeevy companies seems like a good way to avoid 80% of issues kind of like washing your hands can prevent a lot of colds.
No. If it's someone involved in the project for years you feel you can trust then sure, go ahead, but someone coming along being like "sup, commit rights plz" then something's just wrong with you of you agree. If that person were serious about it they could still fork it and maybe msg users of the old version about it.
Sure those people could still mess up by just switching over without checking the new fork, or having good reasons to switch like bug fixes or new features, but at least you as the author of the original lib did your part to prevent any messups.
I do not know if NPM lockfiles are transitive, though. Guess I'd better go looking...
`npm ci` is so new a lot of projects aren't using it and are still using `npm install` everywhere. (To be fair, evaluating if most of my projects should switch to `npm ci` in more places is still on my own personal devops backlog. This thread has reminded me to bump priority on that.)
They didn't know there was a new maintainer, let alone that they didn't trust them.
> In my world, you should have a really great reason to update a third party dependency—“there’s a new version” is not a sufficiently good reason.
Because new versions come out with bugfixes _and_ security patches all the time. In my world (which I'll be clear is not npm), I update all my dependencies to whatever backwards-compat latest version there is, to get advantage of any bugfixes or security patches without having to spend time being aware of every possibly applicable bugfix or security patch. If I spend a day hunting down a bug that ended up having already been fixed in an upstream dependency and I wouldn't have even been subject to it if I had updated... that was a wasted day.
Also if I regularly update, I get security patches without having to have spent time being aware of every one.
This all applies to indirect dependencies (dependency of a dependency) along with direct ones too.
Some of these issues are general to any kind of dependency system. But the particular balance of trade-offs changes in different ecosystems. One thing that seems to be somewhat special in npm is how _small_ and _numerous_ the dependencies are. Npm ecosystem is built upon a large number of very small and discrete dependencies. This has advantages, but the big disadvantage that it becomes very difficult to 'manually' keep track of them, and easy for a bad actor to sneak something bad in.
Security patches are part of my original point regarding why it is bad for a project to die. I have no data on this, but I have a feeling that there are a lot more unpatched security bugs in widely used dead projects than there are popular projects that are taken over by a malicious maintainer.
The simple fact of the matter is that you are responsible for all the open source code you run. If you allow the code to be updated automatically you are abdicating your responsibility to review that code as a trade for being relieved of the responsibility of keeping up to date with patches. You are the one liable when something goes wrong because you are the one who made that decision and misplaced that trust. That is an inherent agreement you make when you use open source code and is often spelled out explicitly in the license.
I don't really understand what you are trying to suggest.
Yes, whether we realize it or not, we are trusting the maintainers of our dependencies. Sometimes that trust is misplaced. That might lead us to try and avoid a particular dependency or maintainer in the future when it happens. And then it'll happen again.
We've built an ecosystem around using open source dependencies. Isn't that what the open source dream was? You might not like it, but not liking it doesn't make it realistic to develop software for money without using all sorts of open source dependencies for which you can't personally vouch for every line of code. I'm seriously not sure what it is you are trying to advocate for.
>1. Pay the maintainers!! Only depend on modules that you know are definitely maintained!
>2. When you depend on something, you should take part in maintaining it.
You can't expect someone to maintain your code for you without you contributing anything in return and if something does go wrong in that situation you have no one to blame but yourself.
For example of I have a project with 5 dependencies, this explodes to hundreds of tiny projects. These individual projects are almost always just a few lines of code, and there's no way anyone's going to become a donor for them.
Many major projects are well funded (webpack, etc), but all it takes is one of those tiny packages to send a malware.
You only need to worry about the level immediately below yours.
Is a change between maintainers a semver-major breaking change? Several people have suggested that that is a baseline that npm could easily automate/enforce. That would have at least sent a community signal to re-review/re-audit the package in light of a new maintainer.
Other small ways that npm controls versioning is that it does not allow you to publish the same version number for a package [2], and `npm version` [3] is often the most common tool for incrementing version numbers, which itself provides a number of semver-focused shortcuts.
So yes, it's certainly a common expectation in npm that packages follow semver, largely due directly to npm's documentation and tools support.
> If you are operating from the premise that the maintainer is potentially compromised, why would you trust them to stick to the semantic versioning spec?
The suggestion was that in the current case the maintainer changed with no warning. One warning system that npm provides is semver major breaking changes. npm does have enough version control (for instance, the part not allowing previously submitted version numbers to be reused) that they could theoretically force all new versions submitted after a maintainer change to jump a semver major version. That would at least send a signal to the large number of developers that don't pay attention to the changelogs of minor and patch versions (and may naturally have a ^ or ~ scope in their package.json) to at least check the changelog for breaking changes. That's possibly the easiest "sufficient" fix for this problem of an otherwise unannounced maintainer change.
[1] https://docs.npmjs.com/about-semantic-versioning
Your world seems more sane, and I (also not in node, JavaScript, npm etc.) generally follow that too. Updating third party dependencies should be a rare thing, something you do only when you critically need to. If I were working on any sort of serious or commercial project, I'd expect to do my due diligence when considering updating a dependency, including examining the dependency's downstream dependencies. Do I really require the additional functionality or fixes from the new version? How mature/tested is the new version? Have there been any changes in the API? What do the release notes say? Are the trade-offs of updating worth it?
Just saying "YOLO, update my dependencies and go!" would give me severe anxiety.
I don't work much with npm. I work mostly with ruby, using bundler for dependency management.
And i update my dependencies to new patch releases _all the time_ without reviewing the releases individually. I think most other ruby devs do too. My understanding of the intention of "semantic versioning" is to _allow_ you to do that.
My projects have dozens if not hundreds of dependencies -- don't forget indirect dependencies. (I think npm-based projects usually have hundreds+). How could I possibly review every new patch release of any of these dependencies every time they come out? I'd never get any software written. If I never updated my dependencies anwyay, I'd be subjecting my users to bugs that had been fixed in patch releases, and wasting my time debugging things that had already been fixed in patch releases.
This is how contemporary software development using open source dependencies works, for many many devs.
By updating all the time you can address bugs and changes as simple fixes here and there. Wait too long and you run the risk of having a really painful merge/update cycle, most likely happening under duress because now you have to update because of some bug fix or security issue.
The more dependencies you have, the more critical it is to stay up-to-date, and thus this directly leads to all the craziness with NPM based projects whenever something goes wrong.
Maybe, but Node had that issue too, no? Remember leftpad?
So, lots of people in the Node ecosystem don't agree with you.
He also feels that there's nothing anyone can do about it except scramble to control the damage. I believe this is currently true.
As quoted elsewhere in this thread
" THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
Making business decisions on the hope that someone else's moral codes will perfectly align with your own is unscalable. That's why we have written laws, codes and contracts.
Seriously. Having a moral discussion in tandem with scale seems very poorly misaligned with the interests, needs, risks of pretty much everyone involved.
A car is merely a fungible vehicle the customer would have been no better or worse off had the robber been driving a different car.
This would be an apt analogy for just giving / selling a code base.
Had it been distributed under a new account/name users could have decided to trust or not trust a new maintainer.
The dev allowed new people to trade under his name and rep worse allowed new people to delegate further to unknown others.
He is morally liable and ought to have known better.
In your analogy the business would be performing the chores for free and telling the users that the business is not responsible for any damage related to the access granted by the key. I don't think most people would sign up for that without a business relationship.
This implies the seller _knows_ the one they are handing over the keys to is indeed a criminal. In that case it is certainly morally problematic.
I see your point but I still think calling the maintainer's behaviour immoral is going a bit too far. Perhaps careless. Or maybe naive. But not more than that.
This is the myth we keep telling each other but I don't seriously believe this is how open source works in reality.
Google says "Definition of Liable: responsible by law; legally answerable".
If you claim he's not legally responsible but is "morally liable", where "liable" itself means "legally responsible", what in your world does the term "morally liable" mean, specifically? What does it mean you can do to him, or what does it mean you should do in future in response to this?
If "he is morally responsible" leads only to "he should feel bad" and nothing more, then what does it matter if he is/isn't morally responsible?
Ok he (does/doesn't) feel (justifiably/unjustifiably) bad .. now what?
Selling a car = shutting down the maintenance of current project, pointing to a fork done by someone.
What happened is just handing the car's keys to someone, without much notice.
I think the largest gripe here is that the original maintainer let the new, unknown maintainer commit to his repo and publish under the already established package name instead of making him fork it and publish as a new package.
But this trust part seems to work pretty well. You need to be trusted to be a Debian package manager, and I volunteer as a Drupal code review admin where we require all contributors to have a real name, and there is a back and forth discussion for a few days until we mark the user as vetted.
If he transfers it knowing other people are going to use it, and don't tell then them, then they cut the brakes, that's a problem. It's not just that it was sold, but people continue to use it and weren't told. That's a different situation.
The legal outcome could vary widely state to state and nation to nation.
If putting a blurb in a text file makes you feel safe about being sued for negligence you haven't considered all possible venues.
Here is an article about liability waivers
It's a bold assertion, I'd be interested to see if you can provide any case law where it's been tested.
I'd be interested to hear if such precedent existed.
So, while I don't think the upthread claim was about the maintainer having legal responsibility, I don't think it would be entirely wrong, even with the license text, if it did.
The idea that the party most proximate to end users is exposed to potential liability is true, but the idea that this must mean noone else in the chain is exposed is not.
If that were the case, you'd pretty much wipe out the software industry as it stands today :)
I'd be very interested in case law where you can see the users of a service (which might not even disclose what software they use) are able to sue the author of a package used as part of that service.
Where the type of harm that results is reasonably foreseeable and could have been prevented by reasonable care by the developer (or maintainer; different though often co-occurring roles), I don't see how the general law of negligence doesn't fit. AFAIK, negligence has no open source software escape hatch.
Really you think that's realistic, given the astonishingly heavy presence of open source software?
Second, since software engineering is not a licensed profession, for any related conduct to be seen as negligent, it has to be something that a reasonable person should be able to avoid and foresee that could cause specific harm. Even a relatively gross act of incompetence by any reasonable engineering standards likely does not meet this bar, given that there's no license required for someone to be in this situation and that it takes a lot of expertise to understand how specific bad practices could cause harm.
I'm not talking about the original maintainer (who didn't introduce malicious code intentionally), but the person he turned it over to (who seems to have).
Legal liability and ethical responsibility are not always the same thing, although it's generally only the first that matters in court.
But the point that I was referring to is any suggestion that the repo. owner who handed it over could bear any liability for doing so, I'd suggest that's not probable/practicable.
Put it this way, I would not suggest relying on that defence in court.
Therefore the issue isn't the license, it is the rules of the law in question under which the author is being sued.
Therefore your intent can matter. Whether a valid contract exists matters. Whether I can be expected to have read it matters. THAT THE INDEMNITY IS WRITTEN IN ALL CAPS MATTERS. (I'm not making that up - see https://law.stackexchange.com/questions/18207/in-contracts-w... to see that it does matter.)
The result? The indemnity in the contract can say whatever it wants and still only provides partial protection. The real rules are complicated and elsewhere in the legal system.
That clearly absolves me of any responsibility, does it not?
This is also very similar to bad entities obtaining or acquiring browser extensions to discretely poison with spyware and advertising, which happened a lot of times.
It's literally not, though. That's what the license says. It expressly disclaims any such thing.
If you want somebody to incur responsibility, you have to get them to take it on. You can't just demand it of them.
Fortunately, there is a good way to do exactly this. Did you bring your wallet?
That's why the idea of commercial support exists. You need to depend on it? Pay for it.
He's a good guy and a great developer, and he's always been supportive of others who want to help out. When working on Scuttlebutt[2] I got some flak for a change I made and he said this[3]:
> I argued against this before, but @christianbundy wanted to do it, so I figured okay do it then. Maybe it is annoying for everyone (in which case christian probably learns for experience) or maybe it turns out to be a good idea! in which case we all do. Either way, shooting down* something someone wants to do (that doesn't have a strong measurable reason) leads to a permission based / authority driven culture. And that isn't what I want. This may make ssb less stable... but I'd rather have less stability but more people who feel like they can work on it / are committed to it.
>* "shooting down" is not the same as negotiating a better solution that addresses everyone's needs. I think that's a good thing, plus being good at negotiating is a very good skill.
[1]: https://www.npmjs.com/~dominictarr [2]: https://scuttlebutt.nz [3]: http://viewer.scuttlebot.io/%25%2B2WIaJ%2BoRVURoTAke9YWuJzp%...
Might be worth looking over some of these to see if he has also given access of them to some dodgy people. As most of them appear to be as old as event-stream and are also probably as unused by him. And if we are concerned, we should, he would advise, simply make a fork and write an email to npm to warn them. But hes a good guy, so don't blame him if you find that he's done this before in the other 422 packages.
Edits: I mean I do understand having software that's old and unmaintained, and it's fine and easy to hand off control to others - we developers are generally trusting of other developers. I wonder if he has also trusted others in this way before and that he has been exploited before without anyone knowing.
Well... yeah. If you don't trust him, don't trust him.
If you don't trust NPM's vetting, don't trust NPM's vetting.
For a successful counter-example, one can look at the well-structured, hierarchical organization behind Linux. Lieutenants gain authority based on the merit of their contributions, and are responsible for reviewing the work of other developers. Authority works, has worked for thousands of years, and will continue to work for thousands more.
Maybe, but the sort of people who are not jerks (i.e. who don't get too much pleasure out of having power) who are willing to lead for free are... very rare.
Fundamentally, non-commercial open-source needs to evolve organizational methods that require less leadership, just because there aren't a lot of good leaders willing to work for free.
Define failure. I don’t know Dominic and I haven’t looked into the Scuttlebutt project beyond being aware of its existence and what it is, but...
He talks about creating a community where anyone is welcome to contribute.
It is perfectly fine for an open source project to have the development process and the community as its raison d'être.
Just because a project doesn’t outcompete every single alternative doesn’t mean it’s failed.
Just because a project isn’t even used by more than a handful of people doesn’t mean it’s failed.
It all depends on what the goal of the project was in the first place, and what the goal of the contributors are.
Yes and in this case, that was exactly the problem.
> It is perfectly fine for an open source project to have the development process and the community as its raison d'être.
Conway's law is not an instruction manual.
When a package gets 2m downloads a day and it still takes 2 and a half months to find a problem, a huge number of developers have failed to do their part.
Anything else is gravy.
I'm actually not aware of a single piece of software that didn't at some point have a production security vulnerability.
Facebook failed.
Google failed.
Amazon failed.
If this is an indictment of anything, it's an indictment of the entire NPM ecosystem -- it's been the wild wild west for years; haphazardly using whatever NPM install gives you is baked into the culture.
Sure, Dominic is an active participant in that culture but it seems to me that it is impossible to have a largely unmoderated volunteer system with as many packages are actively used without things like this happening.
Keep in mind, this is a case where the system worked, more or less -- an observant user caught the issue, and made a public issue of it. Who knows how many packages have slipped by like this?
And also important: It even has a giant number of paid maintainers, for who this is their main job.
For those the incentive to continuously maintain things is different than for someone who gets nothing expect more work out of it.
Linux didn't always have a giant user base, and it wouldn't have gotten there without strong leadership having a sense of pride and responsibility.
In fact, if you had the energy to do the attack at all here (which took some work), having to fake trustworthiness doesn't require much more effort. Just look like a super enthusiastic contributor, put work into the readme, bike-shed over some issues every month, and bam.
Meritocracy is an outdated discriminatory practice.
https://www.theguardian.com/commentisfree/2017/mar/20/merito...
The extent and congruence of the findings regarding, e.g., blinded vs non-blinded resumes alone should put the lie to the idea that a true "meritocracy" is something humans can actually meaningfully do at this point.
Imperfect realization doesn't mean the ideal is invalid (except when that imperfection is inherent to the ideal). You can argue for more equal, more meritocratic community; tearing it down because it's not already perfect is disingenuous and destructive.
Of course merit is judged subjectively (like anything else), but what exactly is the alternative?
In particular, I don't find anything actionable in that manifesto in regards to decision making.
I think that's the key here - organization/hierarchy is important as a project scales up, but he didn't want to stifle a new contribution without good reason.
It was situation where former authority was not interested in being authority anymore, since former authority gained nothing from it and had other work.
He did that too actually.
> you can easily have a GitHub repo for your NPM package with benign code in the GH repo and completely different, malicious code in the package
This is one of the biggest issues that NPM has, along with not enforcing packages to be signed. If the package was signed this would not be an issue as people would see that the signer changed.
Sadly he didn't deserved that trust.
You don't have to feel guilty to abandon the projects. Most us do it, every day.
I'm not against handing over projects, but some vetting has to be done.
1. - don't transfer - mark your repo Abandoned, tell people to use that other person's fork if you have to 2. - does that person have any other online presence? Long running? 3. - does he also put his face in front of the crowd? Talk at conferences?
This creates accountability - someone like that wouldn't pull out the same kind of injection, because he could be caught for that and his "brand" would be destroyed.
I placed an open call to find a new maintainer months ago, and have received many requests. Every single request I've received has been roughly as shady as the request sent to event-stream. I've declined them all due to their shadiness. I know exactly what could happen (this situation right here) if I give it to the wrong person.
For outsiders, it looks like the maintainer is greedy to keep the project to themselves. The bug reports pile up, and dozens of people are offerring to maintain. And then insults start to come up, because it looks like you don't want to hand it over.
If you want to take over a project, you just earn the trust if the current maintainer, be it patches for existing PRs, finding security vuons, etc over the time. Linux has gotten this right.
> maintains hundreds of packages
> hundreds
How can a single human reasonably and responsably do this? This number alone demonstrates how sloppy and inept the Node.js community's practices are.
Not to that degree.
Unix frequently espouses "do one thing", but how much "one thing" is is always open to interpretation.
And it's never been quite as small as e.g. left-pad. It's more something like `printf` or `cat`, which do "print a formatted string" and "concatenate files" as their "one thing", respectively.
Let me reiterate: The problem isn't "single responsibility", the problem is "single responsibility for half a thing".
The pieces are just too tiny, which leads to an explosion of packages, which leads to packages being badly maintained.
I think the big difference is that very few people set their systems to trust and receive updates of `cat` directly from the developers of `cat` (and same for all the other standard tiny utilities). Instead, they rely on a Linux distribution to vet the developers' code and to regularly pull in updates. Maybe there's room for a similar model of distributions vetting code in the npm ecosystem.
https://github.com/freebsd/freebsd/blob/master/bin/cat/cat.c
Also, cat is kind of an important primitive for building a functional system that uses files as one of its core abstractions. It makes total sense that it would (a) exist and (b) be well maintained by an authoritative and reliable source.
Left pad it is not.
Interestingly, the `raw_cat()` function is 37 lines long.
https://github.com/freebsd/freebsd/blob/e511976db9f51bae1a59...
Looking at the kernel itself you see a hierarchy where people are only responsible for small segments of the kernel.
Looking at a standard distro like Debian, which makes this easily available at https://www.debian.org/devel/people you'll see that most individuals are only responsible for a handfull of packages, with the bulk having a team responsible for them.
Not saying that I think maintaining hundreds sounds like a good idea, just trying to point out the misunderstanding :) .
For children, that's much of the reason for Child Protective Services to exist: to regulate orphanages and adoption agencies such that they will thoroughly vet prospective adoptive parents; and to establish and regulate a fostering system—pool of known-good temporary guardians (foster parents) that can take care of children temporarily when no vetted-as-good outsider can be found to take them more permanently.
Now imagine a software org that acts on abandonware software projects the way a CPS-regulated orphanage acts on abandoned children. Start by picturing a regular software foundation like the ASF, but then 1. since all the software is currently abandonware, the foundation itself steps in and installs temporary maintainers; and 2. since the foundation's presence in the project is temporary, they seek to find (and vet!) a new long-term maintainership to replace the existing one.
Of course, that level of effort is only necessary if you care about project continuity. If it's fine for the project to die and then be reborn later under new maintainership, you can just let it die.
No, not if the project becomes malicious. I'd rather it died and I switched to an alternative I can trust.
I don't disagree with your overall premise: if you aren't actively maintaining a project for what-ever reason (you don't currently have the interest in it compared to other project or can't justify the time that you could spend elsewhere instead) and someone else does have the time and the interest, the project should be allowed to live on through new maintainers.
But: just handing over admin access of a project that many rely on to an unknown entity is not a safe move as this case proves. I understand the point of not wanting a permissions based community and so forth, but (and call be cynical if you will) that is rather naive. The world is just too full or arseholes for that sort of idealism to be at all safe in practise.
Instead let the new person/people create a fork and update your documentation to state that you are not currently actively maintaining your fork and people should consider moving to the new one instead. This way no one else unknowingly uses the new fork. Of course people might blindly switch over without verifying the new maintainers which puts those people at the same risk, but at least they take action to blindly move over rather than not knowing at all that it has happened, and people who are more sensibly cautious will hopefully monitor the changes more carefully than they would under the previous stewardship so this sort of backdoor is less likely to go unnoticed.
E.g. I can trust the people behind git: some of them may be poor UX designers, but they won't let out a version that eats my files, or is backwards-incompatible. If a completely different team forked git, I would be quite wary to use the fork outright for my daily work.
When all long-term maintainers stop maintaining a project, the last person leaving should turn off the light. The project should be officially stopped, because no people remain in behind it.
Whoever wants to pick up the development should fork, and use a different name. This way, the forked project will accumulate its own reputation: of course, initially propped in major ways by the idea of "this is a fork of a well-known project".
At the very least, it would be a heads-up for everyone who depend on that project that a major change in it is happening.
I'd also like to see something like StavrosK mentioned in his comment[1] about https://codeshelter.co made a part of this. When a maintainer gets an email for a long-dormant project of theirs, the maintainer needs options. One of those should probably be to yield the package back to the community. A "code shelter" is one way of doing that.
Then the question of, "How do we vet maintainers at scale?" comes up. All I really know is it'd take a financial & human capital investment in Node community infrastructure to make it happen.
I think it's in Node's best interest to do so. These highly prolific maintainers like dominictarr are prime targets for black hats. Overworked, underpaid, huge product portfolio they manage. Who among us wouldn't be grateful for the interest & help?
So Node should invest in fixing this.
It's like litter. It is not realistic to have anyone clean up everyone's trash. It's also not realistic to expect that things remain clean if everyone only picks up their own trash. Everyone needs to clean up their own trash and a little bit more, to compensate for the burps in the system.
If the package remains relevant, someone will eventually fork it. The burden of trust is no longer on your shoulders.
Volunteer project ownership is voluntary, and transferring to another volunteer is the only choice other than abandonment for a lot of volunteers. Package repositories don’t support monetization and there’s no pool of volunteers associated with the repository itself to take up maintenance of what otherwise would have been abandoned.
Would we have preferred if the original creator had simply deleted the repository altogether? The last time someone did that on NPM, it generated deafening howls of rage — but here we are today in the non-abandonment scenario, listening to renewed howls of outrage.
Your statement contradicts itself: there is a pool of volunteers, — they are the ones maintaining NPM packages to begin with. In fact, the author of event-stream has given control to another person, because he believed, that the guy was an ordinary volunteer like himself. Unfortunately, the reality is a bit more complicated than just that: people will voluntarily help you to maintain your projects, but only if they share your goals. Ideally, one should ensure, that the to-be-maintainer has invested a lot of effort in the project they are trying to take over — reported issues, fixed bugs, added features etc. for considerable amount of time. In other words — ensure, that they can contribute to the project in significant way and move it forward. Otherwise, what is the point of transferring maintenance?!
https://twitter.com/andrestaltz/status/1067157915398746114
Many, many people treat open-source projects like consumer products. Except the paying-money part, of course. That they're happy to leave out.
This is a systemic problem, and blaming one guy won't solve anything. Especially since so many are blaming him for not doing enough free work to help them with their paid jobs. If open-source libraries are truly valuable, we need to find a way to jointly pay maintainers for their labor. I can think of many options (a non-profit build with hefty donations from tech companies, governments giving out tax-funded grants, places like NPM and GitHub making crowdfunding 10x easier). But the ideas aren't the problem. It's getting one or more of them done.
To me, this linr of argument completely misses the point.
It's entirely immaterial how a FLOSS project is treated. This problem is essentially an identity hijacking problem. The community trusted the old maintainer, but then he screwed up by enabling an attacker to essentially take over his identity and thus create and exploit a major vulnerability in his behalf.
I also think the words "community" and "trust" are being badly abused here. A community is a relationship of mutuality. Most of the people affected here never did a thing. They were leeches, not participants. A healthy community would have looked at the guy laboring overtime for free and said, "Hey, buddy, let's split that load up." And most of the people who benefited from his work didn't even know his name. They didn't trust him; they trusted the system.
He owed those people nothing. He gave them more than that: a good-faith effort to find a new maintainer.
This isn't the first time that has happened — the story with Google Chrome extensions being sold to hackers should have tought NPM, composer etc. a lesson. Maybe someone should finally sue them to drive the point home?
no. What does "a project dying" even mean? The code still runs, and will for the foreseeable future. If it was half complete, then that would be bad, but this was clearly a complete solution. Let it remain complete. This was a quick way on dominics part to kill the project actually
People who use your project place their trust in you. If you pass that trust on to another developer without your users' involvement, then that developer's actions reflect on your reputation. Why? Because you've not given me the option of only trusting you. Your policy made it a package deal.
It's kinda like how I can't tell Bob about anyone's Christmas gift. He sometimes tells Fred the Loudmouth, who tells everyone and ruins the surprise. Bob never ruins the surprise directly, and he always asks Fred to keep it a secret, but it doesn't matter. I still can't trust Bob, even if Bob's only mistake is that he trusts in the wrong people.
In the end, I built something to "solve" this, a project called Code Shelter[1]. It's a community of vetted maintainers that want to help maintain abandoned FOSS projects, so if you want your project to live on but don't want to do it yourself, you just add it there and you can be reasonably sure that nothing like this will happen to it.
Of course, you have to get a large enough pool of trusted maintainers that someone will be interested in each project, but it's still better than blindly adding people. I can't fault the maintainer of that project, since trusting people is usually fine, but it's too bad this happened to him. The commenters are acting a bit entitled too, but I guess it's the tradeoff of activity vs security.
With Code Shelter you don't have to move the project anywhere, you just give repo admin access to the app and the app can add/remove maintainers as required.
There's obviously a corrective component as well, where maintainers who don't do a good job are removed, but this hasn't happened yet so it's not clear how it will be handled.
Do you mind explaining this vetting process a little more? How can we be sure that something like this flatmap thing doesn't happen on codeshelter?
Trust is definitely an issue here, and trust is something you build, so I hope we'll be able to build enough trust to let people not think twice about adding their projects.
Besides, projects don't usually go from active to completely unmaintained. Adding it to the Code Shelter is a nice way to solve this when you see development slow down, because you basically have nothing to lose.
The insidious threat is that these bad practices will start leaking out of Javascript world and infect other communities. Fortunately most of them are not as broken by default as JS, and have blessed standard libraries that aren't so anemic, so there is less for this type of activity to glom on to.
Unless you're willing to meticulously scrape through every bit of code you work with you're at risk. Even if you can, what about the OS software? How about the software running on the chipset? This is exactly why no one in their right mind should be advocating for electronic voting. There's simply no way to mitigate this problem completely.
With most programming languages, a small-to-medium project might pull dependencies from tens of entities, but with npm, even a small project can easily rely on hundreds or even thousands of entities.
It is easier to weed out unreputable entities when you are depending on fewer entities.
In an app store, apps are self-contained packages with low to no dependency on other apps in the store, meaning that a single compromised or malicious app can only really affect that app's users. The OS may also isolate apps from one another at runtime, further limiting the amount of damage such an app can do (barring any OS bugs).
On the other hand, NPM packages depend on a veritable forest of dependencies, motivated by the "micropackaging" approach. Look at any sizable software package with `npm list`. For example, `npm` itself on my computer has over 700 direct and indirect dependencies, 380 of which are unique. That's bonkers - it means that in order to use npm safely, I have to trust that not a single one of those 380 unique dependencies has been hijacked in some way. When I update npm, I'm trusting that the npm maintainers, or the maintainers of their dependencies, haven't blindly pulled in a compromised "security update". And `npm` is in no way unique here in terms of dependency count.
So this problem is limited to NPM, as far as the potential impact of a single compromised package goes.
If you aren’t reviewing the diffs of your dependencies when you update them, you’re trusting random strangers on the Internet to run code on your systems.
Espionage often spans multi-year timelines of preparation and trust building. No lesser solution will ever be sufficient to protect you. Either read the diffs, or pay someone like RedHat to do so and hope that you can trust them.
The only way to be truly safe from this attack vector is to own all of your dependencies, and nobody is willing to do that so we're all assuming some amount of risk.
If you’ve considered the problem and decided to trust them as a compromise to reduce staffing and costs, that’s a fine outcome in my book.
How many health/medical startups are blindly trusting dependencies today, having never thought through the harm I describe?
Part of bringing in a dependency is bringing in the responsibility for verifying it's not obviously being used badly. One of the things I've come to respect the Go community for is its belief that dependencies are more expensive that most developers currently realize, and so generally library authors try to minimize dependencies. Our build systems make it very easy to technically bring in lots of dependencies, but are not currently assisting us in maintaining them properly very often. (In their defense, it is not entirely clear to me what the latter would even mean at an implementation level. I have some vague ideas, but nothing solid enough to complain about not having when even I don't know what it is I want exactly.)
I've definitely pulled some things in that pulled in ~10 other dependencies, but after inspection, they were generally all very reasonable. I've never pulled a Go library and gotten 250 dependencies pulled in transitively, which seems to be perfectly normal in the JS world.
I won't deny I haven't auditing every single line of every single dependency... but I do look at every incoming patch when I update. It's part of the job. (And I have actually looked at the innards of a fairly significant number of the dependencies.) It's actually not that hard... malicious code tends to stick out like a sore thumb. Not always [1], but the vast majority of the time. In static languages, you see things like network activity happening where it shouldn't, and in things like JS, the obfuscation attempts themselves have a pretty obvious pattern to them (big honking random-looking string, fed to a variety of strange decoding functions and ultimately evaluated, very stereotypical look to it).
And let me underline the point that there's a lot of tooling right now that actively assists you into getting into trouble on this front, but doesn't do much to help you hold the line. I'm not blaming end developers 100%. Communities have some work here to be done too.
Fine-grain sharing gracefully at scale, is a hard technical and social challenge. A harder challenge than CPAN faced, and addressed so imperfectly. But whereas the Perl community was forced to struggle over years to build it's own infrastructure - purpose-built infrastructure - node.js was able to take a different path. A story goes, that node.js almost didn't get an npm, but for someone's suggestion "don't be python" (which struggled for years). It built a minimum-viable database, and leaned heavily on github. The community didn't develop the same focus on, and control over, its own communal infrastructure tooling. And now faces the completely unsurprising costs of that tradeoff. Arguably behind time, due to community structure and governance challenges.
Let's imagine you were creating a powerful new language. Having paragraph- and line-granularity community sharing could well be a worthwhile goal. Features like multiple dispatch and dependent types and DSLs and collaborative compilation... could permit far finer-grain sharing than even the node.js ecosystem manages. But you would never think npm plus github sufficient infrastructure to support it. Except perhaps in some early community-bootstrap phase.
A language that doesn't have a decent standard library means that you'll have to use huge amounts of code that random strangers used, and the chain of dependencies will grow larger and larger.
In languages like Ruby and Python, you have a decent standard library, and then big libraries and frameworks that are maintained by the community, like Rails, Django, SqlAlchemy, Numpy, etc. That's healthy because it minimises or zeros the amount of small libraries maintained by a single guy, thus maximising the amount of code that you can trust (because you can trust the devs and development process of a popular library backed by a foundation or with many contributors).
With Node, almost every function comes from a different package. And there's no bar to entry, and no checks.
If Node.js is going to stay, someone needs to take on the responsability of forming a project, that belongs to an NGO or something, where the more popular libraries are merged and fused into a standard library, like that of Python's. Personally, I'm not touching it until then.
My maven projects' dependencies are technically all pinned, but updates are just a "mvn versions:use-latest-releases" away. But, crucially, I have a file that lists which GPG key IDs I trust to publish which artifacts. If the maintainer changes, the new maintainer will sign with their key instead, my builds will (configurably) fail, and I can review and decide whether I want to trust the new maintainer or not.
Of course, NPM maintainers steadfastly refuse to implement any kind of code signing support...
IIRC rubygems support package signing but almost no-one uses it, so it's effectively useless.
We're seeing the same pattern again with Docker. They added support for signing (content trust) but unfortunately it's not at all designed for the use case of packages downloaded from Docker hub, so it's adoption has been poor.
Unfortunately the development world doesn't really have the same opporunities.
If, for example, npm started to get strict about managing, curating, security libs, they could just move to a new package manager.
Security features (e.g. package signing, package curation) have not been prioritised by developers, so they aren't widely provided.
also you can't transfer ownership without giving away your domain or github account. But you can add others to also upload to your name, but if an accident occurs your liable, too.
I know, it doesn't make much sense why would anyone do that, but then again, I think "why would you that?!" feeling is part of what is triggering the negative reactions here. We just don't expect people to do things we wouldn't.
Quite possibly. But I'd make a conscious decision to do it, and certainly wouldn't be in any position to blame the original maintainer.
[1]: https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.htmlTake a look at the underhanded c contest for plenty of proof where even very senior developers told up front that there is a backdoor in the code often can't find it! And they can't all be blamed on C being C, many of them would work the same in any language.
I don't know the solution, but shaming users and developers for not reviewing code enough sure as hell isn't it.
All that being said, reviewing changes in dependencies is still a good idea, as it can catch many other things.
What I find shameful is the lack of advisory warnings about this risk — by the repository, by the language’s community, by the teaching material.
This should have been a clearly-known risk. Instead, it was a surprise. The shame here falls on the NPM community as a whole failing to educate its users on the risks inherent in NPM, not the individual authors and users of NPM modules.
I mean that's pretty much how the world works. Even running Linux is trusting random strangers on the internet. Most of the time it works pretty well, but obviously it's not perfect. Even the largest companies in the world get caught with security issues from open source packages (remember Heartbleed?).
When I install an app on my iPhone, it is very very hard for that app to compromise my phone or my private data.
In both of these cases, I can download and run almost any code, and be fairly confident that it won't significantly harm me. Why? Because they're extremely locked down and sandboxed with a high degree of isolation. On the other hand, if I install random software on my desktop computer, or random packages off NPM, I don't have such safety any more.
The prevalence of app stores and the Web itself speaks to the fact that it _is_ possible to trust random strangers without opening yourself up to a big security risk.
Edit: partially because of design decisions around package permissions.
What you're suggesting is a great idea from a security perspective. But for typical workflows for JS development it just isn't practical.
Now, maybe this means we need different workflows and less dependencies. But it's so ingrain I don't know that it's easy to fix / change.
If anything maybe those who depend on unchecked code so willingly have the burden of responsibility?
My point here is less about payment guaranteeing a lack of bugs or having someone to point a finger at and more at incentivizing the team building it to fix things quickly. Of course, there's always the <Insert Negligent, Well-Compensated Fortune 500 Company Here> problem, but that's a case-by-case issue.
Supply chain attacks are increasingly being seen as a good vector and there are thousands of unmaintianed Open source libraries buried in the dependencies of popular projects.
Also worth noting that package security solutions, which check for known vulnerabilities, are unlikely to solve this kind of problem unless they track "known good" package versions and alert when things change.
https://github.com/dominictarr/event-stream/issues/116#issue...
I believe npm to be a giant landmine of untrustwothy packages thrown together. No package is trustyworthy as the dependency chain runs too wide and too deep. And having unwieldy dependency trees is considered a feature of npm.
Most programming languages have this issue to one degree or another, which is that they have a package management repository where there is no curation of content, and generally no package signing (there are some cases where that exists, but the take-up isn't always great).
The challenge is that this now seems to be embedded in the culture of a lot of development, and changing that culture would be very challenging, I'd imagine.
The concept is significantly older, but the implementation of a systematic facility for it rather than more ad hoc mechanisms may date from then. (It's a lot harder to sneak in and use an exploit in books of source code used in the 1970s-80s, though; convenience comes with a price.)
Just do the same thing we have for SSH: on first connection (installation), ask if you trust the other side. That goes into your authorized_keys (lock file). If you connect again (update) and the key changed, print a big scary warning. No blockchain needed.
There was a "human error" (Domenic trusting the wrong person) that was completely unpreventable by technological means.
There was no hijacking involved.
The gatekeeper model is a proven one, be it an organisation like debian, a paid curator like redhat or a locked down ecosystem like iOS.
Is it the Github-ization of package creation and maintenance? Is it the pervasiveness of left-pad style packages? Is it npm's governance? Is it the community's perspective and culture? Have the malicious just not made it to other repo systems yet?
For instance, if I was going to do something similar to left-pad in a .NET way, I'd have to create a whole Visual Studio project and build a DLL for a single function, which would rightly be recognized as pants-on-head.
I think it'd make sense for npm to be this authority. They're a for-profit organization who benefits most from the success of the npm ecosystem. So why don't they try and take over the most popular projects like left-pad and event-stream? Especially the projects that are small and mostly unmaintained.
A nasty combination of all of those, probably some more.
I was very excited with Node.js when I first heard about it here on HN years ago. I enjoyed watching Dahl's presentation. But as soon as the crowd that went "it's JS, I can document.getElementsByClassName(...), so I can code servers" flocked in and npm became a thing, that was the end of Node.js as a nice, bright little thing.
I don't want to defend or encourage gatekeeping, but Node.js brought in lots of folk that did not know what they were doing into software development, and they did not stop to learn a thing or two before creating a monstrosity like npm or making things like leftpad and this possible and popular. And there wasn't any seasoned leaders like Python or Perl communities had, so all that happened with no authoritative opposition. If it was Node.lua or Node.pl, the world would've been a different place today for the software development community.
Goes on to defend and encourage gatekeeping.
If gatekeeping is bad, not checking who's passing through those gates when they're wide open is worse.
2) The language -- Javascript has relatively few convenience features in the standard library (at least compared to ruby, which is my point of reference) so more of those features are implemented in userland (and thus imported from npm). I think this is the biggest cause of massive dependency trees -- things like left_pad simply exist natively in many other languages.
3) The ecosystem -- for whatever reason, installing floating package versions was the default for a long time; floating package versions are obviously an order of magnitude more vulnerable to an attacker publishing a malicious update.
Edit: 4) I think it's also probably true that Javascript apps are particularly susceptible to this class of vulnerability, but that doesn't necessarily mean they're less secure overall. Javascript teams tend to move quickly and stay on the cutting edge, so they install lots of packages/updates and are thus vulnerable to malicious updates. Equifax's Java app was compromised for exactly the opposite reason -- they didn't install a security update to Apache Struts for months. That class of vulnerability is far less likely to happen on a fast-moving Javascript app. It's all a tradeoff; the important takeaway is to understand the failure modes for your particular stack and be vigilant against them (which in this case means using a lockfile and doing your best to audit new packages/updates).
[1] https://insights.stackoverflow.com/survey/2018#most-popular-...
Also the low-effort nature of javascript language itself - no strong typing, able to rip out or replace foreign methods, even redefine standard behavior, no private methods, ... if somebody enjoys living in such environment, then they are more likely to also be ok with unmaintained packages, npm bumping your package versions without asking, npm allowing anyone to re-take a package name after it was deleted, developers transferring ownership without as much as googling for the username... but hey, as long as they have cool emoji icons...
https://github.com/right9ctrl/node-scrypt
I would be very suspicious of that as well and audit anything that library has touched.
https://www.npmjs.com/package/scrypt
With one maintainer, Barry Steyn, whose referenced repo is here:
https://github.com/barrysteyn/node-scrypt
Somebody would have to pick it up from a git URL, I think. (Maybe they could typosquat, but we'd have to find that first.)
https://github.com/dominictarr/event-stream/issues/116#issue...
We may need reproducible builds and reproducible minification. If we want developers to audit their own dependencies, in case we deem that practical, packages cannot ship their own minified sources. Auditing the non-minified source is hard enough.
We may need (for-profit) organizations that audit packages and award their official seal which you can trust before you add or update a dependency.
We may need better standard libraries and fewer micro-packages.
I don't agree with this comment, really. But -- there's almost always room for improvement w/tools. IIUC this unpacks encrypted JS bytecode and then executes it? Is there any way to statically know whether this could happen?
"var newModule = new module.constructor;"
...
newModule['_compile']()
From what terribly little I know about Javascript it looks like it would be hard to create a lint/warning that finds this sort of thing without false positives. However, an audit tool that merely discloses this and any other things-that-deserve-greater-scrutiny would be valuable. If you could walk your dependency tree and see whether/when this changes, it would be very useful. Of course, to be useful you probably need some kind of dependency pinning too, which IIUC npm does not (yet) support?The rest of it is extremely dodgy, of course.
However, if you were to audit your dependencies and find a new case of this, you could examine it further to determine whether it passes muster. I suspect code like this is infrequently added to packages.
I think you'd have more luck looking at static strings and warning on any string that has a sufficiently high entropy. Sure, some of them will be hashes for verification, or compressed harmless data, but it it seems like it would be rare enough to be auditable.
If you have any security requirements whatsoever, and you are using node, you need to do your own audits of the (hundreds of) packages you use. If that’s not practical then node isn’t practical
Linux package managers (by and large) have a maintenance process.
Programming language package managers, by and large, do not
So this could happen on rubygems, PyPi, Nuget, etc
This is potentially the biggest Bitcoin heist since MTGox and the biggest news in Bitcoin all year. The last time BitPay got scammed they lost 5,000 BTC and that was back in 2015. I wouldn't be surprised if this is causing the current mass sell-off. Does anyone know how to find real-time transaction volume stats broken down by client, country, etc.?
Then again, the fact that nobody has done this already suggests that it wouldn't work.
But I will say that indeed almost all viable businesses involve some amount of schlep. But just because something involves a lot of schlep doesn't make it a viable business.
Feel free to email me, jt3@justin-taft.com
I've personally met Dominic and chatted with him in the past (although I doubt he would remember me) and he's a great guy. I may not agree with his world views but he's respectable.
Just giving away a repo, and reading his description of it, sounds like something he would do, haha. I'm sure from his view, he's just been open sourcing experimental modules (however big or small) and that's all it is: fun
On the flipside, the inhouse CSS framework where I work actually pulls in event-stream as a sub-sub-subdependency which could have bitten us possibly if the attack wasn't so specific.
It's easily to blame Dominic but ultimately, he's just a guy who made some code for fun and people started using it.
I'm sure when you're in a position like that, one of two things is likely:
1) You either don't look at download/usage analytics because why would you? It's one of many projects and you're not measuring their success by anything more than how you feel. People can use whatever code but it's not exactly a product being sold
or
2) You do look at the numbers perhaps but 2 million people? How do you even quantify that? It could be 1 million bots, 2 million hobbyists, 50,000 enterprises who don't cache internally and npm install over and over, or even all 3 combined!
Anyway, this whole event hasn't happened in a vacuum so at least it'll (hopefully) spark constructive discussion about the state of JS and companies/projects relying on possibly unmaintained sub-sub-sub... you know
A lot of very common packages...
But even 5 years ago I remember thinking "400 dependencies for a project is just WAY too many" and imagining something like this. And now I see some of the big frameworks require literally thousands of packages. Is there something about Javascript that necessitates this kind of ultra-fine hairsplitting on packages, or could that be made better?
also languages like python require an affirmed compile time resolved call to the classes so it can more easily know what code is not used or not going to ever be used.
Javascripts string based introspection and runtime generation of code being the backbone to the entire framework also makes it impossible to know what parts of included code are even used, just because no code calls out to it at build time doesn't mean the code isn't going to generate calls to it.
Coming from other languages, the lack of built-in functions seems crazy, and the NPM situation just weird. An empty template I use for bootstrapping jquery-based web apps, based on using Gulp 4 for transpiling ES6, SASS compilation etc, has a node_modules folder with 23,268 Files across 3,080 Folders!
"he emailed me and said he wanted to maintain the module, so I gave it to him. I don't get any thing from maintaining this module, and I don't even use it anymore, and havn't for years."
in the thread. Then, I looked through the commits and realized how not ready @right9ctrl was to maintain a heavily used package. Basic linting and small, iterative changes are absent.
So, when I see Dominic-さん just "handing" the package over -- it seems careless. This is often why -- I try to write my own packages unless I absolutely need some sort of framework. It's trading one set of problems for another (the potentiality for bugs vs. a package as a security issue).
npm_package_description = 'A Secure Bitcoin Wallet';Far from being ironic, this is the normal functioning of the system.
I looked at the linked commits: https://github.com/dominictarr/event-stream/commits/master
and it looks so surprisingly normal.
I am pretty sure I wouldn't have caught it even if I was code reviewing it.
Did someone heroically rerun every package's build process to see if the minified output was tampered with?
Just wondering out loud on how to detect even one of the other active exploits that are surely in the wild. This one was live for 3-4 months before it got found by luck?
For example I would really appreciate if they added something to detect/warn when a package needs eval or http requests permissions (yes, I know npm packages don't have a concept of permissions).
Anyway your commonly repeated advice on how to defend yourself isn't practical (short of not using npm). The dependency that got me was nodemon. That ought to be "trusted code" with 17k github stars, 1 million downloads per week but apparently not. As I said already, even if I did check the commits to all my dependencies, I don't think I would've caught this one. It's a one line change in the minified build file of a transitive dependency nested 4 deep. All the commits around it are very normal looking.
Hopefully you see why your advice is similarly impractical.
This is not just a security risk. There might be some business implications if somebody would sneak in for example some AGPL licensed software to the dependency chain of a popular package with more permissive license.
Morals and ethics.
This is free code, responsibility to check may need to have a social shift to those who use it.
I feel like 90% of the open source libraries I've written are maintained by someone other than me now. If that's considered unethical, next time I'll just not open-source it.
To your point, if the author added the malicious code, I don't think the license would necessarily protect him. Intent is key. But he didn't.
It's an author/maintainer issue (and not an NPM issue) because the maintainer explicitly gave the bad actor the right to publish via NPM's delivery channel. The maintainer did so in gross negligence, in my opinion.
For that reason, I think you are wrong.
I think if you want to stop working on a project, and it's on github, just leave a note in the README, archive it and move on with your life. Don't give the keys to someone on the internet.
What JS/NPM needs is some sort of certification program for secure libraries that would fill the void of the lack of a standard library. NPM of course would give a warning when installing a non certified library.
This would be a tremendous effort, but it would really be the only way for NPM to survive in the long term. NPM is one of the reasons we've moved to Go for our server side code.
It's one of the reasons, but far from the only reason. With Go, Python, Rust, Swift... is there any reason to use Javascript on the backend? Most attempts I see to "share" code between the browser and the server end up adding needless layers of abstraction and complexity.
Another one is making a GraphQL API that concentrates data for the clients from different places. It could be made in another language, but it's better if front end developers maintain that project.
For bite sized cloud functions Node is also not a bad choice.
But for pure backend projects, yeah, Node isn't the greatest choice these days.
Who, what, where, why, and how need to be answered.
Usually when there's a link like this on hacker news, one of the highly moderated comments explains the situation in ways that don't require as much context as a typical commiter would have.
For example, if this had been @dominictarr/event-stream the entire time, and he chose a new maintainer, that would then by definition change the name of the package to @right9ctrl/event-stream, requiring all users to intentionally change their dependency.
Yes, it's painful. But it prevents getting caught out by a malicious patch release.
That said, the whole github/npm relationship is broken to begin with. No simple mechanism exists that I'm aware of to verify that a tag as pushed to GitHub is exactly the same as what is published to npm. And due to various prepublish hooks, it can be very difficult to actually prove this. One solution would be GitHub support for building & publishing packages, with a Travis-like build log, so users can be assured that the package as it sits on npm actually was built from expected source.
@right9ctrl could have just avoided committing this entirely, and just published a malicious patch release. If he had done that for a few days then published another that removes the injection, it might have gone unnoticed for months.
Except you can build packages using one of a million different tools. Unless you could only publish packages that were built using specific vetted tools (and vetted plugins for those tools), that wouldn't change anything. If I can change anything in the build pipeline, I can control the output.
Something out there depends on ps-tree (directly or indirectly) and the description in its package.json is the AES key for this string.
npm_package_description = 'A Secure Bitcoin Wallet';[1] https://hackernoon.com/im-harvesting-credit-card-numbers-and...
Look, if you're pulling external dependencies into your project from anywhere, and you don't version pin and hash compare those dependencies every time you build, then you get ZERO security guarantees. Simple as that. You might have some "in a perfect friendly world" expectations, but no guarantees, no recourse and no one else to blame when the world does not conform.
The notion that the author of the package could/should do anything to mitigate that for you is ridiculous. Ignoring many, many plausible coercion/honest mistake scenarios, just consider that a once-reputable maintainer can over time become malicious. That's it. If you have any actual real security requirements, then I'm sorry but you don't get to say "but but but DRY!!!" and call it a day. You are responsible for what software you run in your product.
I'd also say NPM's centralized naming structure is also problematic, people don't want to update 1000's of modules to some new latest version of X which was forked and is now maintained, so it's handed off like this.
I'm interested in who noticed the compromise, when, and how. Because with open source, noticing the compromise is about the _best_ we can expect from a "worst case".
Good security scanning and perhaps identity verification on Github could help. So at least you can ignore pull requests from authors that didn't verify their identity.
Honestly, a problem like this is overdue, especially for the NPM ecosystem with its propensity for a huge number of transitive dependencies, but also for all other major repos (nuget, cargo, CPAN, Docker, etc.). OS-based repos (Debian, Ubuntu, FreeBSD) might feel a little safer because it's harder to become a publisher, but it's not at all impossible. Perhaps the only reason it hasn't been a target before now is because there are easier avenues for cybercrime.
If you think that ownership transfer should exist at all, then the attack vector still exists no matter how long you wait to trust right9ctrl.
There are a number of factors that could be considered when giving somebody this kind of responsibility, including existing contributions to open source, contributions to the project at hand, and public profile. As far as I can tell, "right9ctrl" had none of these.
This maintainer actively ceded control of his library to some random person.
https://github.com/GovAuCSU/DRAT
Disclaimer: I am not a dev, just a pentester by trade so the code is probably ugly as fuck for many of you =))
The practice of handing control of open-source packages to new maintainers is old and well-established. You can go to Sourceforge and request to take control of any old, low-impact, unmaintained repository and Sourceforge administration will likely grant it to you after a considerable delay and some investigation (at least they used to do so in the past, not sure if they will continue to after all this mess). The responsibility to ensure, that new versions of packages haven't been subverted by malicious actors, have always lied with people, who brought those packages into distribution — the package maintainers. Unfortunately, NPM does not have any "maintainers" in the traditional sense — package authors can't be trusted to remain impartial, and "administrators" just sit on their backs, waiting for devs to fill their repository with quality software. There is zero oversight — you can do anything with your packages as long as it does not outright contradict local law, including openly selling them to hackers, openly incorporating backdoors, and even sabotaging entire package ecosystem by unilaterally deleting hundreds of popular packages.
Covert transfer of control should not even be possible in centralized repository like NPM. If you want to give your package to someone, that act should be registered with repository administration, and future users of package should be warned of it — just like users of services and goods are commonly informed when an existing company changes it's organizational structure. It is one thing to privately give access to Github repo to someone. It is entirely different thing to hand over a repository package — which is automatically distributed onto large number of computers around the world. In later case authors, who failed to announce the change of ownership to repository maintainers, should bear full monetary responsibility for their actions.
The golden days of including any old gem or npm package that seems to solve the problem you're looking at are over.
npm i --dry-run event-streamThis could be solved in a few ways.
Maybe instead of signing over ownership, maybe the protocol should be to force a fork and make notes in the README that there's a new repo whose author has not been vetted and should be used with caution.
(I expect that question is very naive from the point of view of people who know this area, but I'd be interested to be pointed in the right direction.)
Blame yourselves devs, not the author who donated hours and hours of free work for your benefit.
Just mark it as deprecated and call it a day, like a normal and responsible person.
Somebody really needs to explain to me how this works. The dude's bio on github is "antipodean wandering albatross". Nothing against it btw.
Says you! I don't agree with that. Author clearly doesn't either. You're painting it as though the author knowingly handed it over to some hacker when in reality he had 0 cares about this package and just handed it over to the first guy who asked. I would have done the same probably.
If you don't have the care to maintain it or do any sort of vetting, you shouldn't have the care to do anything at all. Literally, leaving this unmaintained would have been a better solution.
So how can you be mad?
Here's a quick summary of the situation.
WHAT HAPPENED?
==============
A widely-used dependency was handed over to a different maintainer, who proceeded to add a malicious sub-dependency to it. The sub-dependency only contained the malicious code in a single release, and only in the minified version, likely to avoid detection.
WHAT DID THE MALICIOUS CODE DO?
===============================
The malicious code used the 'package description' to decrypt its payload; this was done to ensure that the malicious code would only run when the dependency was used in a specific application.
That specific application was the Copay Bitcoin wallet, from BitPay. The malicious code injects itself into the application, steals the user's Bitcoin wallet, and sends it off to a remote server. It's currently unknown who operates this server.
There may be other forks or projects with the same package description, "A secure Bitcoin wallet", that are also affected.
HOW DO I KNOW WHETHER I WAS AFFECTED?
=====================================
As a developer: Look in your `package-lock.json` or `yarn.lock` for an entry of the `flatmap-stream` dependency, specifically version 0.1.1. Unless you were developing on Copay, the code probably wouldn't have run, but you should still remove the dependency.
As a Copay user: No release of Copay included this malicious dependency. It's unclear whether copay-dash, a fork of Copay, did. Contact your wallet developer for more information.
HOW CAN THIS BE PREVENTED IN THE FUTURE?
========================================
This is probably the most crucial question here. There's already a "lol JS" bandwagon gaining steam in many places, but that's not really the issue here; nor are small modules the issue.
Dominic Tarr is an established contributor to the ecosystem and part of the community, not some random unknown party, so this issue would have existed regardless of the size of the dependency.
I would argue that the real problem here is of a social and partly-economic nature: There's no clear way for maintainers to find a good new maintainer for their projects.
This is not a problem that is unique to JS, either; such support structures are missing in most ecosystems.
As a developer community, we should probably have a serious discussion about the security risks introduced by lack of funding, and by the lack of a support structure for no-longer-maintained packages.
The latter can be solved on a grassroots level, without any payment being involved. The former is going to involve funding models of some sort, and I hope that this discussion can be had in a neutral manner, not as a marketing pitch for a specific startup.
Another important problem is that there's essentially no code review tooling. While in this specific case even a review would have been unlikely to catch the issue, it's pretty much impossible right now to review all of your dependencies in a project (in any language) without going crazy.
Possible solutions to that would include a review tracking system, that integrates with your package management and flags any new dependency in the tree as 'needs review' before accepting it.
- An ecosystem of massive amounts of transitive dependencies increases the number of people you need to trust. If I wanted to attack a project that used NPM, their package.json dependencies would be a really good place to start. Find the least popular transitive dep they use and email the owner to see if you can be a contributor (repeat for all of their xdeps). If they don't immediately give you publishing rights like OP did, then show some chutzpah and make valid commits until they do. While this attack works on any programming language's dep system, it's easier the more transitive deps a project has. People ITT blaming the OP don't understand this attack always works on a long enough timescale. Do you think there isn't someone out there who would make high quality contributions to an xdep of primedice.com (online gambling site) for 5 years to finally get publish access?
- Anything may run during `npm install`. npm install supports an --ignore-scripts argument to not run any scripts during install. This should be the default.
- Unqualified module names make it more desirable to "take over" a package than just publish your own package "npm install <username>/event-stream", so it contributes to an ecosystem of ownership-transfer that's far less likely to exist on, say, https://package.elm-lang.org/ where everything is qualified by a Github username.
- NPM website doesn't show you source code. The github link on the project page is just a convention. I think the NPM website should have a light source code browser of whatever is in the tarball that you download and execute during `npm install`. Bonus points for reproducible builds from that source.
- Developers don't actually review every bit of code they use and execute, especially not transitive deps. And we certainly aren't going to bother to download the tarball from NPM and unpack it to inspect the code of every dep. Most people reading this don't even know how to do that.
I've thought of some ideas to help the situation, like creating a Github shield that verifies that a conventional build script like `npm run publish-build` reproduces the tarballed code on NPM, but then I would just be doing free work for the NPM organization, and it's still just a hack.
NPM modules don't even have source code on display. Someone has to download and check the tarball before npm install.
Also, Elm packages are qualified by a github username so there isn't an ecosystem of ownership transfer. No juicy name squatting. People just fork.
Finally, don't forget that my point is "there are a few issues with NPM that make this kind of thing especially easy/lucrative". That's a far cry from "everything else is bullet-proof" but it's tempting to argue with me as if I'm saying that.
I'm a little dismayed by the great number of comments I had to read before someone pointed this out...
If you insist on pushing out hundreds of packages on npm, then drop support for them and hand over rights to random strangers, you are acting completely irresponsibly. You shouldn't be given a pass because you're doing it all voluntarily. Your professional reputation should suffer for it.
If these huge companies profiting off OSS work actually contributed financially and with time, maybe maintainers would happily remain maintaining.
If you have this "no warranty, no responsibility" attitude then how can you have pressure to maintain anything? How does that translate into pressure to hand off the project to strangers? Nobody asked for that.
> Don't contribute to OSS and you have no problems.
Yes, please don't contribute if you have this kind of attitude. It's harmful. If you can't maintain the package, deprecate it. Don't hand it over to an unvetted entity. If we can just agree on that M.O. we'll have a much better situation.
> If these huge companies profiting off OSS work actually contributed financially and with time, maybe maintainers would happily remain maintaining.
Ifs and buts and sugar and nuts. They don't contribute, they never have and they probably never will, you are responsible for your packages even if you are doing it voluntarily.
It's stupid to blindly trust someone's code, regardless of who they are of if they are the current maintainer, that is not how you build secure software.
If you want to start some organization which vets people for maintainers go for it, but don't expect maintainers to do it, I can guarantee you that thousands of maintainers do not. You're responsible for what ends up on your servers.
Two options:
1. Stop supporting the package, mark it as deprecated, ignore/delete the spam
2. Hand over project to a stranger, endangering all of your users
One of these options is irresponsible, the other one isn't. I'm not asking anyone to do something for me, I'm asking them to not do something for the sake of sanity.
> It's stupid to blindly trust someone's code, regardless of who they are of if they are the current maintainer, that is not how you build secure software.
I agree, but both of us know that pretty much the entire Javascript ecosystem is exactly that stupid. Let me ask you: What have you personally done to vet your dependencies? Have you used Babel or any of the other popular Javascript packages that have a huge dependency tree? If you have, chances are you are effectively blindly trusting all the developers in that tree. You're not checking every single commit that goes into it.
> If you want to start some organization which vets people for maintainers go for it, but don't expect maintainers to do it, I can guarantee you that thousands of maintainers do not.
I don't expect that they do, I expect that if they fuck up that their reputation takes a hit. That would some incentive for not fucking up. Instead, there isn't even any consensus that this guy fucked up. He's taking no responsibility whatsoever.
Node, a general purpose language, has a scope that's much, much larger.
"Oops, I just gave the repo to an unkown dude", OK. Sure. Just replace "unkown dude" with "to my colleagues at 5 eyes secret service organization"
This kind of mistake is not a mistake, not from a dude like dominic.
I suspect Dominic didn't see himself as the maintainer of the package, but as a maintainer of a copy of the package (like how git is designed to work). With this perspective, of course someone else can be a maintainer.
The problem is that NPM changed their upstream from Dominic to right9ctrl without checking. (Maybe because Dominic clicked a button — NPM is still responsible for accepting that decision or not.)
The centralized naming scheme is to blame here, it normalizes these "cute" names and users have demanded these since day one. They could/should have probably been scoped from the beginning, or people should just stop trying to capture fancy names.
I get tired of receiving emails about those kinds of things personally, you try to just ignore them, sometimes their request sounds reasonable but still, can those people be trusted, who knows.
It would have been better just to use URLs, so if you see "npm install https://github.com/dominictarr/event-stream" then it's clear you're trusting "github.com".
(Yes, DNS is centralised. That can be fixed independently.)
Sorry, could you please explain what this means? I keep seeing comments about bumping version numbers (e.g. the right9ctrl did some shenanigans with versioning).
I ignored this particular request via email a few times, because I just really don't want to spend time dealing with Node stuff. You then get a request from NPM support asking the same thing, so I agreed to it, that's all it takes to get access to a package really.
I think from now on I'm just going to tell people to come up with a different name, this wouldn't be a problem if the names were scoped by default.
Even if I take your sentence dead seriously - the code injection, the crypto wallet stealing, the disruption to the downlevel users, none of that matters only Dominic's credibility matters - where does that get me?
Assume that Dominic was bribed to inject the code himself, with a share of the profits, and he did, maliciously and criminally, and he gets arrested, and people were still disrupted and money was stolen and insecurities were introduced and flamewars were had .. and his reputation is now justly and fairly tarnished for all time..
what good does that do anyone? How does that avoid it happening in future or change how we respond to it?
This is a community problem based on insufficient incentives and the way that the software development community is content to allow individual labor to replace community efforts and you, as a member of that community, are kinda pissing in the pool right now. If you can look past the initial dismay, this is a time for the software development community to look in the mirror, not to pick up your torches. It's a project that he hasn't touched in years. Why should he care? What are you doing to make him want to?
And somehow, despite this experience, I am also blessed with the minimal capablility to understand the swamp-ass social toxicity that's happening here. I am likewise blessed with the modicum of good sense to realize that this is not a problem that goes away by stuffing one person in the wicker man and setting it ablaze. Systemic problems have systemic solutions and a real part of actual security is applying systemic solutions instead of looking for scapegoats that let you think you're a tough guy on the Internet.
Because it is about feelings. It's always about feelings when somebody fumfuhs about how serious and important it is that they, or a likeminded compatriot, get to be an asshole to an individual about a systemic problem. It's just about their feelings and given place of pride because they're theirs. So maybe we shouldn't play that game at all and maybe we should look at the systemic issues that actually matter.
Haveaniceday.
So who were you trusting?
What's the risk for Scuttlebutt, Dat and the Knight Foundation? Dominic will be the only maintainer of a project and he'll foolishly hand over maintainership to a government spook?