So the first step of this huge mess was: a social engineering attack. Attacking a tired, burnt-out open source project developer and peer pressuring him into giving more control of the repo to the attacker.
But important software needs to be identified and proportionally more scrutinized by multiple independent parties, that's the lesson. Identification is the hard part. You can't easily determine that half of the world relies on this particular piece of software, or that it enables access to desirable targets.
Yes, this almost succeeded... but can you imagine how many scenarios where someone such as Andres Freund would have found irregularities, but then.. what? Just had to report it to some webpage's contact page? Without being able to even dig further?
Would he have known what was happening, or would it have just ended up as an oddity, with no source code, and with the binary purposefully obscured, and so on?
Or... even worse, it's reported directly to the 2 guy team, and the guy who put the back door in... takes the bug report?!
From where I sit the sleeper/mole problem exists in companies, and can be far harder to detect.
Yes, you get more eyes and people like Andres Freund.
However, if this had been a mole in a company, he wouldn't be able to hide behind a possibly anonymous fake persona and (likely) be immune from any consequences/fallout from this attack. It would be harder to gain entry in the first place, he would have needed a real identity. Background checks may not be a big hurdle, but they're at least something more than signing up for a GitHub account. He also wouldn't have had his posse of anonymous sock puppet accounts to add pressure to the original maintainer.
I also think that just being able to ramp up on liblzma and its dependencies to undertake this effort in the first place is a huge head start vs. trying to execute the same attack on a closed source corporate product.
At the same time, there are probably lower hanging fruits to attack if you really do get a mole inside a company, in addition to there being fewer eyes/opportunities for the attacks to be discovered as you pointed out.
I honestly don't know how all of these factors add up. I expect the argument opposite to yours to be raised (again) in the wake of this incident. I'd personally be hesitant to raise this argument (not that it matters here on HN).
You can save a lot of time reading John LeCarre novels and take a short summary like this one of Adam Curtis [0].
If you watched the recent Oppenheimer film you'll know the name Klaus Fuchs. But what about Guy Burgess, Kim Philby and Anthony Blunt? If MI5, MI6 and GCHQ are. almost by tradition stacked to the rafters with defectors and spies, enclaves of enemies within, and enemies within enclaves of enemies... how does anyone expect a commercial company motivated by money and with such a weak perimeter as a "job market", to do better?
Trust does not have an organisational solution.
[0] https://www.bbc.co.uk/blogs/adamcurtis/entries/3662a707-0af9...
Lots of companies hire remote workers sight unseen, and not all require proof of identity, and where they do, that proof can possibly be easily faked by someone willing to break the law.
Larger Open Source projects - e.g. Debian Developers, also have real-identity verification through chain of trust.
> Background checks may not be a big hurdle, but they're at least something more than > signing up for a GitHub account.
GitHub doesn't allow more than one free account per person. Companies might not look for employees sharing an IP address as much as GitHub does.
> He also wouldn't have had his posse of anonymous sock > puppet accounts to add pressure to the original maintainer.
Depending on the software, it might not be that hard to be a customer (or even pretend to be an employee of a known customer, if they aren't validating incoming requests claiming to be from customers enough) and add pressure for features that will provide cover for a backdoor, or even for a product that isn't a commercial success to be handed over to an external developer.
You should check the thread posted yesterday, the Lastpass guy who raised a PR for a go binding for xz but was otherwise unrelated to this fiasco already faced a bit of questioning regarding their motivations from their employer based on a user reporting them from a contact form.
Moreover, many companies already have information from background checks, and in certain countries, they also have the tax identification number of the employee which can pretty much identify who put in the backdoor.
If this happened on a proprietary software project, …I have verified proof of identity documents for everyone on my team. These can of course be faked or fraudulently obtained, but it heightens the barrier to entry and leaves more of a paper trail. That’s not something that we have in this scenario. There’s way more give and take between open and closed source than open source purists make out. They just like the idea of being able to see source code.
Success is making it to some machine, and then it's "how long it's successful for". Who knows how many other such vulnerabilities there are, and the openness of the source didn't protect us as quickly as we thought it would.
Did it? At least in Debian, where it was found in the first place, it only existed in sid/unstable and I think perhaps in testing. It never made it into stable which is the only thing you should be running on a production system.
I think in Fedora it was also only in a testing or RC of the next stable version but I'm not sure. Also unsure about others. I have seen several distros putting out advisories (Gentoo, Arch, OpenSuse and more) but I didn't go into the specifics.
Though it was frighteningly close, 24.04 is scheduled to be released on April 25.
The end goal seems to have been to get it into the LTS releases of rhel and Ubuntu. Since this was a valuable it's probably something kept around for when other methods fail. I doubt it would be used before the really valuable systems are compromised (rhel and Ubuntu in cloud and governments)
There's nothing wrong with Debian stable having years-old packages either. In any case:
> The current "stable" distribution of Debian is version 12, codenamed bookworm. It was initially released as version 12.0 on June 10th, 2023 and its latest update, version 12.5, was released on February 10th, 2024.
https://www.debian.org/releases/
That's just over 1 year. Sure, packages themselves will be older than that since they take some time soaking frozen in testing before they become part of stable but that's generally a good thing.
In any case, if there's pressing need for some newer packages, it's always possible to use apt pinning to pull those on top of a stable base.
> > It made it to production, for a long time
I never said it didn't make it to production. I was asking if that's the case, and giving one data point.
It apparently didn't make it into Ubuntu LTS either, so there's another one.
If you have information that proves or disproves either point, please present it.
That is pretty hard to defend against. I wonder if they were planning this all along, or if they wound up with a greedy plan later, or what.
"I don't feel like it, if it's important to you then feel free to fork".
That's really all that's needed. "I don't feel like it" is all the justification you need.
Some guy just made a compression tool, because some people like doing that kind of thing, or because it was useful for him. He didn't ask to be made "critical infrastructure" or to be responsible for the security of sshd or to have some business depend on it, or anything like that. No one even asked him.
That's really all that's needed.
that's much harder than it sounds.
having someone fork your project can give you the feeling of loosing control over the project as potentially all your users might go with the fork. that fear is often strong enough to push yourself to do things that will avoid a fork.
it's a desire for harmony and a fear of conflict
That's the issue right there. Why would you care?
Clearly, the maintainer is invested in having a "community". Why? They must expect something positive coming out of it, so they are invested in having that community, which then means they have something to lose if that community moves somewhere else and they are left behind.
That's what enables this social exploit. A takeaway can be not to get so invested in having a community. These are users of your software, and their "utility" is in helping find bugs, perhaps suggest improvements or even provide patches, making the thing you as the original author made better. But that can clearly become a burden.
And that doesn't really change that things really are that simple, kind of. How do you stop smoking? By not lighting up any cigarettes. Of course it's not that simple, but ... it also kind of is.
this is an important observation.
a change of culture is really what is needed here, because it implies that not only maintainers change their behavior, but everyone involved, and the question is not so much what any individual can do for themselves, but how we can help others with that change and spread it
For the most part, the people who care already care (you, me, most others commenting here) are not really the problem.
So ultimately the only thing you can do as a maintainer is set your own boundaries for yourself. And the original message in that thread ("there is no activity, what are your plans?") is a fair question, although the follow-ups are not.
All of this is true for a lot of the internet by the way. The best thing of the internet is that anyone on the planet can talk to anyone else on the planet. When I was a kid I did some ham radio with scouting, and you could talk to people from places like Russia and the US. Wow!
The worst thing of the internet is that anyone on the planet can talk to anyone else on the planet. You're constantly exposed to a seemingly never-ending stream of assholes and idiots, and a single asshole can ruin the day of dozens, hundreds, or even thousands of people every day. Maybe this is just 1% of people, but 1% of 8 billion is 80 million.
being nice to each other is just one of the many things that we as humans need to work on. and we are working on it, and hopefully some day we will be able to achieve it.
> Redict is a Finished Product
> Drew is a controversial person
(he's been rude/mean in the past)
xz should be pretty much finished as well, major overhauls like the "ifunc" feature to inject alternate function implementations are not really justified. Beware of busybodies and this whole "the community demands vibrant evolution of the project" thing. xz does its job already!
And being rude/mean is not a plus, it's a minus, but it does seem to correlate strongly with leading a high-quality project. Even if it's not the best way, this person has the guts to say no in unequivocal terms. You might just have to accept the downside of a rude/mean maintainer as a common unwanted side-effect of an effective maintainer. (Linus Torvalds also comes to mind of course.)
If someone had slapped me down for being an idiot I'd have most definitely found something else to spend my free time on.
Today, I'm not super great at coding but can hunt down segfaults like a truffle pig and submit bug reports (with code) that demonstrates the exact issue if it's something I can't figure out on my own.
Yes, that makes sense.
> ifuncs are an important part of that.
No, IFUNCs have absolutely zero performance benefit over regular function pointers for internal functions. If anything, they can limit optimization oppertunities the compiler has with other approaches. The only benefit IFUNCs bring is being able to avoid another indirection when replacing already exported library functions in their entirety. That's what they are there for - different optimized implementations for things like memcpy in glibc.
"I feel a huge responsibility to please every user who reports a problem, shortcoming, or asks for a feature, which I need to address ASAP" is one attitude.
"I work on it whenever I feel like it, and if you don't like that then I don't care" is another.
Those are on the extreme end and for most people it's somewhere in-between.
It's very common for people doing volunteer work to lean too much towards the first. They have trouble saying "no" and bite off more they can chew, even without direct pressure. Learning when to say "no" and setting boundaries for yourself is absolutely a vital skill for any kind of volunteer work – without it sooner or later you will burn out.
When I was a scout leader burnout due to this was a major cause of attrition. We made it very clear and very explicit there was no pressure for anyone to do anything they didn't want to, and that there was no shame in not wanting to do something just because you didn't feel like it. But it still happened, because people still feel this pressure, even when it doesn't actually exist.
The reason I ask is maybe the people who do are (in general) self-selected from the subgroup of people who don't just say "sod off" when someone is rude or inconsiderate.
If that was their stance in life, they would likely not put up with maintaining a popular open source project for very long.
And sure, I understand why people feel a responsibility. And it's fine to take this responsibility too. I'm just saying: there is no need to.
The entire point of this Free Software/Open Source is to give people the freedom to do whatever you want with some piece of software, without having to be beholden to the original author. That's pretty much the entire point.
Anyone in the world can be a maintainer for xz. By forking it and applying useful patches.
It was definitely another world. I do agree that maintainership issues were already there, but I think they were smaller in number. Between the explosion of projects and the explosion of users, they are now on a different scale.
The others I don't know, but MySpace was not a particularly big operation. Before WordPress, most sites started as PHP at some point were expected to migrate to something else, because maintainability of PHP3/4 projects was a big challenge (hence the "shame"). Facebook was the first company that simply refused to do that, and focused on improving PHP instead.
What PHP lacked and Ruby had was rails like framework , the team from 37signals singlehandedly shifted the momentum out of PHP with their work on rails.
It was almost 20 years ago, and no, I generally agree with the person you're responding to that OSS has changed significantly in these regards since the early-to-middle 2000s.
I'm not even disagreeing with the rest of your take, just poking at this idea that time hasn't passed and changed things. Some days I look around our industry and feel like it's nowhere near the one I was working in before.
The peanut gallery of non-contributors are only peers in the sense that they pretend to speak on behalf of some OSS community. And the fact that they are spokespersons is by default suspect. The attacker is a peer in the sense of being a contributor. So is he a peer pressurer? Again we come back to the teenager who has to follow the whims of his peers in order to be included. The maintainer is already inside of his own playground. So the pressure to be part of the “community” is really the incredibly abstract thing that the peanut gallery was referring to: you ought to do so-and-so in order to be whatever I think of in my head as an OSS maintainer.
This can be rejected out of hand if you really believe that maintainers don’t owe anyone anything (because of free labor).
But this gets incoherent if you want to assert both of these things:
1. There is no social contract for OSS maintainers: they can toss their PC out of the window and go on a five-year pilgrimage without telling anyone
2. There is some community which has power over the maintainer to peer pressure them
If you really want to double down on (1), the “cure” is what the OP suggested: say no and walk away.
I don't think the maintainer is at fault to any degree here. Sure, this could have been avoided if the maintainer refused to be pressured and kept sitting on the project and letting it die, but it's not his fault that he didn't do that, and I wouldn't want that to be the default for maintainers either.
All the pro-social benefits with a side-dish of the nuclear option. That’s coherent I have to admit.
In that case one can limit one’s interactions to other invested parties, i.e. contributors. Granted then you are still interacting with the attacker but you’re spared from the peanut gallery.
In real life volunteering you don’t get random drive-by input from outsiders. The input (and whatever peer pressure) is only from other invested parties.
> I don't think the maintainer is at fault to any degree here.
I’m having a hard time understanding moral arguments. “Fault” and “blame”. Everyone is condemning the peanut gallery for complaining about passing on the maintainer stick to someone else. Yes, including people who say that he could have just “not got peer pressured”. To be clear it’s not about the maintainer having “fault” or the peanut gallery/the attacker being wrong. Both can have “fault” in different ways. Like, clearly the attacker is the one who did something bad. Now there’s only a question of what other people could have done differently.
> Sure, this could have been avoided if the maintainer refused to be pressured and kept sitting on the project and letting it die, but it's not his fault that he didn't do that, and I wouldn't want that to be the default for maintainers either.
The maintainer could have done something different but he didn’t and that’s not his fault. It seems that we all agree that he had a live option. You just want to not associate it with “fault”.
In another comment[1] I asked what moral obligation a maintainer has to herself. Only to herself.[2] Focusing on that angle seems more fruitful than talking about “fault” in the abstract since that just leads to back and forths about whether people should protect their wallets better or whether or not people should just stop pickpocketing people.
The goal of this subthread seems to be about how maintainers might protect themselves (for their own sake) from this kind of thing. Laying out the options that are in their hands (and not just how the world around them should become better) seems pertinent to the issue.
[1] https://news.ycombinator.com/item?id=39882721
[2] Like asking about whether someone has a moral obligation to eat healthy. It’s not about other people.
Sure you do: everyone likes to comment on whether your volunteering work is an effective use of time and resources or not.
Peer pressure among adults is far more widespread and powerful than the limited peer pressure among teenagers.
I hate this trend of shouting "victim blaming!" once someone tries to explain things or analyse anything. Not everything needs to be a value judgement. "X happened, and Y could have prevented it" is not a judgement.
If I were in the maintainer's shoes, and was feeling ambivalent about handing over maintenance to a fairly unknown person, this kind of social attack would definitely push me over the edge, exactly as it was planned to.
But you can insert "I'm not trying to blame anyone, but here are some suggestions to modify cultural norms so these things are less likely to happen in the future" if you want. Or you can just assume good faith and take that as implied unless demonstrated otherwise.
Also: as far as I'm concerned there is no "peer pressure" here because these people aren't "peers". They're just some random people who, as near as I can tell, have done fuck all. There is not even an attempt to help out. Not even the question on how to help out. These people are supposed to be the maintainer's peers? Yeah nah. They're just shouty entitled internet nobodies that have not even attempted to contribute anything constructive or signal any willingness to do so (not even "I have been using the patch in production for half a year without problems", which would actually be a small but useful way to help out).
When it comes to these types of things you need to accept that you can't change every person in the world; you can only change yourself. If you cycle a lot you better learn to anticipate assholes doing asshole things. Is that fair? No. But it beats being run over and getting hospitalized, or worse.
That’s what I find incoherent about it.
1. The consumers are little ants that have no real power over anybody, certainly not to help the project itself
2. On the other hand they are powerful enough to peer pressure the only person who has the power to drive the project forward
I think their paraphrasing was a fair representation. You want to have it both ways. You said some incendiary victim-blaming thing, but now you're backpedaling with a "no no, you misunderstood".
I think its far easier to make $3,000,000 as a hacker than a worker/entrepreneur.
Its way easier to find flaws/bugs than to do the entire Capitalism thing correctly.
Then I see that half of these major attacks required social engineering.... Maybe being a hacker is significantly easier. I only need to fool 1 person, there are a lot of people, and merit isnt exactly how everyone got to their position.
Anyway point being: People are amazed by hacking, they shouldn't be, its relatively easy if you are a mere 10 year programmer. Most of us pick relatively moral work, so the number of attacks are small. It is also why we really need to treat security on computers like its no stronger than your home's front door lock. There are too many attack vectors to be perfectly safe.
+1
Between roughly 1999 and 2013 I was primarily a test engineer for networking switches/routers/telephony products. I found bugs for a living and wrote them up, and in the process I found plenty of security vulnerabilities and wrote them up. Security bugs aren't really that different from other bugs. Yet for some reason we lionize people who find security bugs.
Most security issues are simply quality issues. But by calling them security issues we shift the focus away from the software producer creating shit code to an attacker doing something bad.
The case with xz is a little different, because we have someone who intentionally added bad code and tried to hide it. But for unintentional bugs that rise to security vulnerabilities it's 99% a QA problem.
The argument around blame shifting is apt. The same case has been made for the usage of the term 'bug' (aka an externality). It's 2024, we don't have moths crawling into relays on our computers. We have implementation faults, invalid designs, unsound architecture, inaccurate documentation, ambiguous requirements, and a myriad of other ways to express how software may be defective.
Using those terms hurts, and may even invoke some level of concern from those outside of the engineering org - this is a great reason to embrace them.
It would be pretty poor QA who only tested happy paths . Any competent quality analyst will be expected to test for failure states, boundary conditions and so on .
And if you do make that $3m as a hacker, now you have to spend your time worrying about whether someone will eventually catch you because of that one time you logged into a service without using enough VPNs.
Even if we say "no payment, no customer," it won't prevent determined attackers from paying significant amounts of laundered money in order to be treated as customers.
In fact, Occam’s razor says that the malicious code was injected by a compromised account and not a malicious actor who spent years steadily getting into position to attack?
The normal commentator who complains is probably not malicious, probably not aware of the pain they might cause, is probably just not even thinking of that angle.
Let's take on face value that it was the Chinese, and that China is communist. I mean, "Kumar" and "Tan"? Maybe it wasn't, but it doesn't matter for my purposes:
They took an overworked peon of the capitalist enemy that provides a ...
... do I even need to expound? well it's fun ...
... collectively and idealistically produced common operating system "for the people"
... that is exploited and neglected by the rich and powerful, to a degree that society, not just computers, overall society operates on this operating system, and the profits from that are hoovered up by the powerful.
The communists attacked the exploited proletariat to get to the enemy. And they are forcing the enemy capitalists to either pay the proletariat properly (they won't) or continue to be vulnerable to the growing communist power in the far east.
That's what this distills so well, within a political/ideological conflict that has now spanned 100 years: capitalism vs communism.
To wit, a proper functioning capitalist system to reward market value for produced value would properly pass compensation to this poor soul, and incentivize others to help him. Barring that, a proper functioning government would recognize the public good of this and provide support to the core infrastructure software to enable other private enterprise to produce tax revenue.
THOSE AREN'T HAPPENING, so the Communists can attack this with impunity and in perpetuity.
Here's the thing folks, we've been living in, as they used to say "uninteresting times". Post-WWII Pax Americana, even the Cold War was basically peace, has been cranking for 80 years now.
People, that is coming to an end:
- Russia / China are destabilizing demographically and simultaneously becoming militant and totalitarian
- Global Warming will ramp up the pressure on populations, food shortages, production
- The US will likely retreat to a more regional focus as production is onshored, regionalized (or at least centralized to our hemisphere)
There's going to be more state conflict, the Ukraine war is just the beginning. The world stakes are rising.
Let’s not.
(I guess in the case of xz, Jia Tan has earned the cred before going rogue; the one-off “maintainer needs to be replaced” campaigners, however, haven’t.)