XZ: Repo maintainer Lasse Collin responding on LKML
lkml.org
lkml.org
Don’t forget that he made xz for free, as a hobby project, and likely got duped by “Jia Tan” same as everybody else did. He’s not obligated to solve this on any particular timeline.
Xz is not a business, so if your business got in trouble because a single solo hobby dev was a bit too trustful, it’s your job, and not his, to mitigate the problem.
Software may have backdoors, I'm not convinced this couldn't have happened if a large company was managing the product. AFAICT, open source software has a better security track record, in general.
IMHO, if it is anyone's job to look for and prevent these kinds of backdoors, it's probably companies like RedHat and Canonical. They bundle the tools with their products and they charge money for support.
The only practical solution is for these companies to rewrite the core packages from scratch, written from the ground up so that they are easy to audit - no unexplained binaries, no arcane build scripts, etc.
iirc Studies was done on this and there was no measurable difference in security issues when it comes to open vs closed source.
Just because the code is visible doesn't mean much, as you need to have the right eyes to actually notice security issues and a lot of open source projects don't have this.
Also closed source projects have the advantage where if they have a massive company behind them, that company has the resources (i.e. $$$) to hire highly specialised people to look the security of the software. Open source projects usually don't have such resources, heartbleed and xz are indicative of that.
Arch, (Gentoo, Nix, Mint, etc.) have no commercial aspect on the other hand and certainly doesn't have the money, but those distros are offered on the same "as is" basis as the downstream software.
Money alone makes poor filler material to patch such holes. Microsoft famously doesn't make their stuff for free, and yet somehow Windows systems have gained this reputation of having holes such that a train can go through.
People build trust, take shortcuts, cut corners all the time. It's how we are wired. Doing otherwise is an energy sink for our bodies. We don't make ourselves overspend energy when we can.
There is software out there that is both much worse and costs in the tens to hundreds of thousands per seat per year.
Software isn't tooth brushes. The simpler and more specialized it is the cheaper it is to maintain. The more complex it is the harder it is to maintain.
I see no problem.
The issue in this case is that Linux distros took it upon themselves to alter its code base by linking in libsystemd (thus also liblzma) for the dubious benefit of better systemd integration, which comes with a generous helping of attack surface.
Nope. With open source the user is responsible. It's been this way from day one. Come to think of it, almost every piece of software I've ever used that is commercial has a no warranties and not suited for purpose clause in the EULA... because all the components the vendor used have that clause in the license to the vendor.
Much of the worst professional service I've ever received, despite paying them a lot of money, has been from the example you gave, doctors.
Doctors aren't alone, of course. The most expensive restaurants I've ever been to have been some of the worst dining experiences I've ever had, while friends or relatives inviting me over for a backyard barbecue have been among the best, for example.
I've also found this to be the case for lawyers, accountants, teachers/professors, mechanics, and other well-compensated "professionals".
Software hasn't been an exception, either. I've had more success with free software than I have had with the equivalent paid offerings many times over.
In general, I've found that service quality is far more tied to the provider's level of passion for the task at hand, rather than anything to do with the amount of financial compensation that may be involved.
>Most of businesses use Linux for their servers mainly Ubuntu/Debian
Businesses should calculate with the risks, and also, use Long Term Support versions of software, not unstable bleeding edge fresh from the oven ones. Which they do. And so they were not vulnerable at all.
Google does that with Project Zero but few companies are wealthy enough to afford that. The way out is economic, not technical: insurance, and mutualizing the cost of security audits. I wrote up my ideas on the subject here:
This is supposed to be what services such as Tidelift provide - companies pay them, and OSS maintainers sign up with them for funding (and other support? unsure). But it's tough to apply this through the whole OSS ecosystem. To me, this should be what Canonical's support contracts should be for. They're perfectly positioned to help mediate these types of issues.
It's worth noting that vanilla upstream OpenSSH DOESN'T depend on xz. The dependency was patched in by the major distros to better integrate with systemd - and it's (AIUI) a transitive dependency from systemd - even the patch doesn't use it, but it gets pulled in with other systemd stuff.
that's what i believed as well, but i didn't take time to verify. and i didn't know about the details. thanks. as you describe it ssh still doesn't depend on xz (and why would it?) so part of the problem here is software architecture.
how is it possible that a seemingly unrelated dependency somewhere within systemd can affect and be exploited through ssh directly?
shouldn't it be possible to keep that separate?
doesn't openssh itself already implement some form of privilege separation?
how does software architecture here and in general need to change to prevent things like this?
i am sure somewhere these questions are already being discussed. i'd appreciate any pointers.
that's the critical question here. it's not that enterprises need to take over maintenance of xz, but that the critical packages and their dependencies need to be audited on a regular basis. the development of xz is ok. what it needs is help with code reviews. and if it can't receive those, then it needs to be removed as a dependency of openssh.
I don’t remember the company names, but I guess people will remember the incidents.
- Solarwinds have been breached end to end.
- A company has been bribed (forced?) to ship backdoored encryption algorithms.
- A network hardware supplier’s firmware had been backdoored by Chinese IIRC.
- NSA backdoored national standards.
- Microsoft has been breached end to end.
In short, even if you’re a company, you’re one NSL, one bad actor, one misstep away from “total pwnage”.
I trust some individuals for developing critical software than entire “enterprise”s.
Everything is as strong as their weakest link.
Using standard libraries for common stuff like compression, cryptography and whatnot is vastly more preferable over everyone shipping their own crypto, or worse, patches of crypto (see the Debian SSH key vulnerability of 2008 for an example [1]). For protocols it's in the end just as bad, it's a nightmare to keep different versions of the same program to be able to talk to each other, but now imagine a literal ton of programs who all have a wild mixture of statically shipped libraries, homegrown stuff that has barely been tested... no, just no. Not a world I'd like to live in.
Why is it unreasonable?
I actually see it as very reasonable. You use a package in a commercial distribution, you use aggregators like Github sponsors to pay the maintainer (a subscription, not a one-off payment!). What's unreasonable about that?
Paying the maintainers is nice, but I think the megacorps and governments has to fund an organization that does security audits of all the sensitive packages, instead of heaping yet another job on the shoulders of the maintainers. It's especially important to not saddle the developers of open source with red tape, most of them would think that getting paid is poor compensation for the lost freedom.
Also, I'm on the same page with you. Block the malicious actor, continue sipping your coffee.
(Though come to think of it, that would make the analogy less accurate, not more, by implying interdependency where there is none.)
Linux Kernel. libc. Compilers. Programming Languages. Big libraries like filesystems, while generally not open source, system ROMs. Tons of network and system RFCs, standards like POSIX.
These are these long, wide, thin slabs you don't see. Others are too high level. These slabs are very close to metal. You're not kneeling enough. :)
I take some of the recent concern from the Rust community about burnout and such to be early stage discovery of the economic problems in maintaining nontrivial infrastructure as a common good.
Think about it this way. Say you are volunteering for a non-profit. You obviously don't have to do it and could just chill at home instead. But once you have agreed to say take a volunteer shift on Saturday, you are obligated to show up. Obviously no one can really stop you from skipping, but it would look bad on you and may get you banned from the non-profit. Along the same token, you obviously shouldn't steal from the non-profit, or say randomly beat up the customers/clients. If you accidentally set the building on fire which ends up killing 5 people, I would also assume you would help in some fashion to clean up.
Just because you are doing things for free does not mean you don't have social responsibility for your actions. You don't have to work, but just saying "oh my software ends up compromising everyone's computer? sorry not my problem" is seriously misunderstanding your role (I'm not saying that Collin is doing that). Similarly, if I just randomly slapped you in the face, I might have done it for free, but you would also probably be pretty pissed at me and expect an apology.
Just want to point out that this isn't a typical "demand underpaid open source developers to work 24/7" issue here. This is a unique circumstance where his action has directly resulted in this happening. I feel bad for him being guilt-tripped to add a maintainer after reading through the old threads two years ago, and as I said I don't want to burn him at the stake, just saying that saying he has no responsibility to help at all is really a bad take.
He has no responsibility to help at all.
All of the work done is that of a donation. Everyone who used xz did so for their benefit. Collin may feel a personal duty to work in the interests of the users of xz, but that's their exclusive choice. Any claim otherwise is on par with saying required-donation.
> "oh my software ends up compromising everyone's computer? sorry not my problem" is seriously misunderstanding your role (I'm not saying that Collin is doing that).
Then why mention it at all? But even if you did want to say that, that's true! You don't blame the victim of some abuse. Everyone, you included (assuming you use xz in anyway) were lied to. Just because Collin was lied to first, and most, doesn't change the fact the person who did all of the important work was also lied to and betrayed.
> Similarly, if I just randomly slapped you in the face, I might have done it for free, but you would also probably be pretty pissed at me and expect an apology.
Personally I'd be neither pissed, nor would I expect an apology. Shitty people are gonna be shit. But your mistake is blaming Collin for 'introducing' you to the person that slapped you in the face. But you're arguing as if Collin should have known this guy was a shitbag, and should hold the icepack while you berate them for making the mistake of trying to help other people... wtf?!
This us vs ourselves toxicity needs to stop. It's not us vs us. It's not even us vs the them. It's us vs the problem, us vs the malicious. The person writing the code is the good guy here.
No it won't. Unless you have obligations.
> Along the same token, you obviously shouldn't steal from the non-profit, or say randomly beat up the customers/clients.
That's unrelated, to say the least. You obviously shouldn't steal from anyone nor beat up anyone, that's not related to this specific case at all.
> Just because you are doing things for free does not mean you don't have social responsibility for your actions.
No. No one forced the society to use the work that somebody was doing for free. And if society decided that it's fine to use it, it's on the society to take the associated risks. Everything else is a good will of the maintainer.
To be exact, he didn't do it specifically, but the non-profit is usually locked but he unlocked the door to let someone in and that person ended up stealing from the non-profit.
I really don't understand why people do not see that if you cause a giant mess, you have some responsibility to clean up after yourself. Again, I'm not saying that they should be obligated to do work. But this isn't "working". This is cleaning up an issue that you have caused.
> No. No one forced the society to use the work that somebody was doing for free. And if society decided that it's fine to use it, it's on the society to take the associated risks. Everything else is a good will of the maintainer.
If I voluntarily baked a cake for you for free because I was nice and I mistakenly poisoned you, would you say "Oh i decided to take the risk of eating your cake anyway, not your fault"?
I didn't have to bake the cake. But since I decided to, I have basic obligations including not putting poison in it.
> because you do something for free doesn't mean your actions don't have consequences.
Of course there are consequences - society is a closed system. Though it doesn’t give you ANY right to demand anything from the person who you didn’t pay to, directly or indirectly. Stop juggling the facts and whining, no one owes you anything, as I already said.
I would say it's definitely the most interesting one we know about so far :)
Dissect "worst backdoor EVAR". Why is it bad?
Is it because thousands of companies and millions of users depend on it?
On one guy on holiday working for free!
When we are essentially in two cold wars currently with ICBM nuclear powers?
Linux is the battleground of cyber warfare, which is being funded by nations probably in the hundreds of billions of dollars. A trillion dollars or more of economic activity runs on it
Windows? Come on, even buggier and less secure, and far less transparent.
Monetary support of Linux maintainers is now a national security concern. You can't have hundred billions for attack and zero dollars for defense, that is a stupid strategy.
He's doing fine, by the way, and mentions that the messages of support are appreciated but not necessary.
He's more focused now on figuring out what happened, how he missed it, and deciding a plan of action for cleaning things up.
(paraphrasing from conversations in the public channels)
Poor guy will go through more stress now even more than that created and imposed on him over 2 years by the attacker.
If anything we should encourage him to look back at his mental health as not being his fault and that we need to protect ourselves.
For what?
Seems like something serious and illegal happened. Possibly to do with a hostile state. Where I live, there would be at least 2 different organisations that would be very interested to find out what happened as quickly as possible. I don't for one second think he was involved but he would be essential in any investigation.
What makes you that confident they would? What if Jia actually works for them?
https://tukaani.org/about.html
All meaningful new information will probably come from identifying Jia Tan, if that ever happens. I haven't done much reading yet, but it seems to me there was no connection between him and Lasse outside the Github project.