Xz: A microcosm of the interactions in open source projects

There's this weird trend that's been happening for some time now that tries to make non-fully-open groups look wrong, but honestly, has anything ever actually been done by such open groups? As far as I can tell, "closed community" is a necessary (but not sufficient) condition for any kind of quality creative output (this includes individual projects as single-person closed group). Having a core of creators surrounded by distinct group of fans and secondary contributors is a natural organization that forms spontaneously.

Engage with people that have earned it in your eyes, whether by contributing to your project via code, assets, bug triage, writing a good and effortful bug report, whatever. Just ignore what the larger internet has to say about you and your creations.

It is necessary to establish clear boundaries of what can and can't be provided by the maintainers. If not done at an earlier stage of the project, the support burden becomes too much to bear at which point the maintainer transfers ownership, and the project suffers from catastrophic consequences such as the xz backdoor we're talking about here, or other cases where the project mostly stalls and serves as an ego-boosting platform for the new maintainer, as was the case with PhantomJS[6] before it was shut down.

This can also happen in your life, where a "friend" sees that you possess a certain skill, and then gradually tries to push an inordinate amount of their personal work related to this field onto you.

Personally, I think it's best to use an approach with extremely clear communication as to what the maintainer can and cannot provide. This can be seen, for example, in yt-dlp[1], where the consumer is clearly informed upfront that not providing detailed information as requested will lead them to block said consumer; or sqlite where their position regarding contributed patches[2] and support[3] is similarly made clear.

Having a shouty BDFL like Torvalds can also help improve code quality[4] and questionable contributions[5], though it is better that the shouty BDFL makes statements that are professional and do not show as much aggression; so for example, "Mauro, shut the fuck up"[7] would become "Mauro, your response is completely unbecoming for a Linux kernel maintainer, and is not in line with the promise of not breaking userspace."

[1] https://github.com/yt-dlp/yt-dlp/issues/new?assignees=&label...

[2] https://www.sqlite.org/copyright.html

[3] https://www.sqlite.org/support.html

[4] https://www.theregister.com/2024/01/29/linux_6_8_rc2/

[5] https://cse.umn.edu/cs/linux-incident

[6] https://github.com/ariya/phantomjs/issues/14541

[7] https://lkml.org/lkml/2012/12/23/75

I mean sure, but how do you intend to stop people? You can't stop them from setting up a forum somewhere.

This is why github discussions is useful though. A feature request in a ticket is something that needs to be handled. A feature request in a discussion is just some users talking about stuff they think would be cool.

I'm not saying the community shouldn't be there, it's just that the barrier to entry is so low, that people can "contribute" with very little effort, that's become a significant problem as Internet usage (and acceptance of shit-posting trolls as just being a cost of anything online), so the signal/noise ratio is awful and we end up in an easy slide to controversy and spite and the good actors wanting to walk away.

I think this is why I don't do social media any more. I think it's why I'm not on the mailing lists I once was, or have interest in discord or a great many open community sites. It's just exhausting. HN, slashdot and some group chats in Signal with people I have known for many years is the closest I get to anything like that any more.

I've thought about doing some blogging without comments and a means to contact me directly another way privately. I've thought about making contributions to other projects with my only conversations being about my own commits, but that doesn't work for leading my own projects if I want to be "nice".

I think leading a project for me might involve _not_ letting others contribute to my version without clearing some arbitrary bar. No mailing list, no discord, no developer community. That doesn't feel optimal on some metrics, but it might be optimal for the metrics I care about.

I’m not much of a gamer at all. But the once in a blue moon where I dip my toes in, it’s not that long before I’m met with a forum post or reddit comment making some technical assertion that sounds incredibly presumptuous if not downright incredibly unlikely.

I believe this masochistic behaviour stems from a combination of both general optimism and dedication to a particular cause. The more that developers are passionate about what they are doing, the more they are willing to engage with the peanut gallery.

I'm not saying everyone need to be Linus Torvalds circa 2012, but I do think more people need to be a bit less precious and sensitive, especially when receiving direct communication about their abilities.

You don't need ego-boosting yes men, you just need to work with more experienced folks, and that's okay. But the problem is that there is an army of people online who will take offense to that, and I don't know what the solution is. Best of luck.

WHO FUCKING GRANDSTANDS ENDLESSLY ABOUT CYBERTHREATS AND THE NEED FOR BETTER SECURITY?

WHO FUCKING SITS ON BILLIONS IF NOT TRILLIONS OF DOLLARS IN FUNDING?

That would be the FUCKING GOVERNMENTS OF THE FIRST WORLD. Whose modern economies increasingly rest on open source to a degree they possibly don't understand.

Yes I understand part of this comes from funding from another part of these governments. Oh well, cat and mouse. Once we wrung our hands over these governments having power over these. Well, these days we are facing far more totalitarian state threats and totalitarian wannabes in the private sector.

Government save me, as the less totalitarian option!

Linux and BSD the operating systems should UNQUESTIONABLY be funded to the degree of multiple billions per year by, I'll just put it, "NATO". Here's the true value of open source in this model: it is the perfect public record, unlike untold other aspects of government output.

You must produce public vetted code in Linux/BSD.

This person should not have been "a person". This should have been 5 people, funded likely at least 10-50k/year for their roles.

The described "hit" by the intel services of course is frighteningly easy. But what is most horrid is that this poor person is being subject to very common abuse that comes from non-state-actor manipulation. The article says it:

"this is where our software comes from" -- this poor dude that is trying to help and being abused by state actors on one hand, and ridden thanklessly and for no monetary or fame benefit by massively deep pocketed corporations, governments, and billionaires.

For all of Linus's famous swearing abilities, he hasn't used it to address this publicly. He should be dressing down all corporations and governments at this point that need Linux. There isn't a "going back" from Linux at this point to some closed source option.

Linus and Linux have massive sway here. They could threaten to stop work on the kernels.

I guess fundamentally, this makes me angry because this is basically bullying the nerd in high school to get his homework/answers. The nerds need to realize their power here.

Other vulnerabilities snuck through the commit chain? I get that, usual spy stuff. They actually kind of messed up here. Usually it is within the "cordiality" of things, just quietly ride the overworked unpaid nerd to benefit. Here they inadvertently exposed the real truth of everything: the hidden contempt we treat these people with as a society.

Needless to say, I did not assist him in any way and closed the issue as wontfix.

I've also had people demand that I expand the narrow scope of my project to cover a whole class of devices with completely different APIs, instead of the sole focus on the series from the same manufacturer of devices I own and use. Closed as wontfix as well with a clear statement that I will not be expanding the scope, but am more than happy to link to their project covering it.

But we all should realize that any interaction also comes with a cost. If you report a problem and don't manage to describe it well it doesn't help the maintainers one tiny bit, in fact it might just create work and a bad feeling.

So my approach is to "just" write stellar bug reports including ways of bisecting where the problem might come from.

By whom? You don't have to accept such expecteations - plenty of projects that just throw code over the wall and do fine.

I think its far easier to make $3,000,000 as a hacker than a worker/entrepreneur.

Its way easier to find flaws/bugs than to do the entire Capitalism thing correctly.

Then I see that half of these major attacks required social engineering.... Maybe being a hacker is significantly easier. I only need to fool 1 person, there are a lot of people, and merit isnt exactly how everyone got to their position.

Anyway point being: People are amazed by hacking, they shouldn't be, its relatively easy if you are a mere 10 year programmer. Most of us pick relatively moral work, so the number of attacks are small. It is also why we really need to treat security on computers like its no stronger than your home's front door lock. There are too many attack vectors to be perfectly safe.

"I don't feel like it, if it's important to you then feel free to fork".

That's really all that's needed. "I don't feel like it" is all the justification you need.

Some guy just made a compression tool, because some people like doing that kind of thing, or because it was useful for him. He didn't ask to be made "critical infrastructure" or to be responsible for the security of sshd or to have some business depend on it, or anything like that. No one even asked him.

(I guess in the case of xz, Jia Tan has earned the cred before going rogue; the one-off “maintainer needs to be replaced” campaigners, however, haven’t.)

In fact, Occam’s razor says that the malicious code was injected by a compromised account and not a malicious actor who spent years steadily getting into position to attack?

Let's take on face value that it was the Chinese, and that China is communist. I mean, "Kumar" and "Tan"? Maybe it wasn't, but it doesn't matter for my purposes:

They took an overworked peon of the capitalist enemy that provides a ...

... do I even need to expound? well it's fun ...

... collectively and idealistically produced common operating system "for the people"

... that is exploited and neglected by the rich and powerful, to a degree that society, not just computers, overall society operates on this operating system, and the profits from that are hoovered up by the powerful.

The communists attacked the exploited proletariat to get to the enemy. And they are forcing the enemy capitalists to either pay the proletariat properly (they won't) or continue to be vulnerable to the growing communist power in the far east.

That's what this distills so well, within a political/ideological conflict that has now spanned 100 years: capitalism vs communism.

To wit, a proper functioning capitalist system to reward market value for produced value would properly pass compensation to this poor soul, and incentivize others to help him. Barring that, a proper functioning government would recognize the public good of this and provide support to the core infrastructure software to enable other private enterprise to produce tax revenue.

THOSE AREN'T HAPPENING, so the Communists can attack this with impunity and in perpetuity.

Here's the thing folks, we've been living in, as they used to say "uninteresting times". Post-WWII Pax Americana, even the Cold War was basically peace, has been cranking for 80 years now.

People, that is coming to an end:

- Russia / China are destabilizing demographically and simultaneously becoming militant and totalitarian

- Global Warming will ramp up the pressure on populations, food shortages, production

- The US will likely retreat to a more regional focus as production is onshored, regionalized (or at least centralized to our hemisphere)

There's going to be more state conflict, the Ukraine war is just the beginning. The world stakes are rising.

Worthwhile noting: It happened in the open, archived for the world to see at any time! The attacker needed to be extra careful not to raise suspicion, both acutely and for accumulated evidence in history. Of course, easy to say "Hindsight is 20/20", but we can probably agree in actual hindsight, we easily see a lot of suspicious acts around the issue. An individual incident may be chance, but as a whole things become clear more easily.

So, I think there is a teachable moment here: When something seems just a tiny bit fishy, do a background check, investigate. You may not catch all something, but you probably won't miss everything.

Now, looking at the level of sophistication and risk sneaking people and code into open infrastructure, imagine chances of getting caught infiltrating a large company writing closed source binary blobs for drivers, firmware, ...

It's not an either/or proposition. I definitely think it was state sponsored, AND one method used was social engineering a burned out maintainer.

like let's say you're the NSA and you know Russia (/china/etc) is trying to do this backdoor. maybe you send Freund an email through one of your cutouts and say hey, I've been looking at the ubuntu RCs and we noticed some performance regression in the postgres tests, etc... do it from some corpo email from a "friend" at some bigtech company that legitimately uses postgres/ubuntu and it's completely 100% deniable and innocuous.

it'd be interesting to see correspondence to/from Freund on his mailing lists too, see if there was anyone that (in retrospect) might have been tipping him down the path of discovery too.

(which is not to diminish in any way what he did... chasing a tiny perf regression in core library functionality back to root-cause is no mean feat. especially when it's code that is actively trying to evade detection - watching for debuggers, etc. Although that heisenbug nature might have also made it more compelling to these sorts of people ;)

That's an excellent mind set when reviewing code, no matter security or not. But especially for security. How could this be wrong? What are the corner cases? How could anyboy break this? What do we need to test? That kind of scrutiny is crucial for keeping the quality of your code base high, no matter who posts the PR.

It is implicit in your statement that you know some "proper" way to think about trust. If it is the case, please do share with us your thoughts on trust.

Laws are highly regulated. The DMV policy manual is, like most government day to day policy, run through “managers” who report to a elected official (ie it can be very well written and considered or made up on the good after a bad newspaper headline)

Anyway I think I am saying there is likely to be a consolidation of FOSS - where we spend a fortune trying to bring it under some “approved and controlled” process.

We can call that trust.

I am not sure.

Because that's an insanely hard problem that's outside our area of expertise.

Consider how much money governments spend on all the red tape they add to increase trust. If there was naturally perfect objective alignment and trust I bet any infrastructure project would cost about 10% of what it does.

In any case, I agree technologist aren't thinking about it because the only way to extract massive amount of wealth is by centralization/monopoly. Hard problem though, the appropriate amount of redundancy is hard to know.

2) A potential explanation for "why now" is that systemd DID prevent these dependencies from loading automatically in a patch one month ago [0], and the patches to lzma enabling the backdoor merged a few days later, followed by (as we know) an immediate and somewhat heavy push to get distros to upgrade driven by sockpuppets. It could be a total coincidence, or it could be that the attacker jumped to pull the trigger before the window of vulnerability started closing on them

[0] https://github.com/systemd/systemd/pull/31550#issuecomment-1...

I think it's a bit cheap to blame systemd here, and systemd does not equate directly to Red Hat either.

1. Might not have the vocabulary to express themselves correctly

2. Not understand the nuance and connotations that their word choices imply

I've seen my colleagues get frustrated in meetings while simulatenously having to assume good intent.

Choosing names like the bad actor(s) did gives them an advantage in this scenario.

Western and Chinese are not the only possibilities, it could be a Russian, or Indian or pretty much anywhere else that engages in intelligence gathering operation.

They also used a sock puppet with a seemingly German name (Hans Jansen). However "Hans" has not been a popular baby name in German speaking countries for many decades. You'd expect any "Hans" to be over 70 or 80 by now. Unless they are American, like Hans Niemann.

This could be a cultural oversight on the attacker's side, which points towards any culture that has a wide spread belief that typical German names are Hans or Fritz. From my experience, that is especially true for English speaking countries, mainly because of stereotypes displayed in Hollywood WWII movies. I wouldn't be surprised if cultural perception of Germany was different in China.

Edit: some clarifications

What would raise my hackles would be someone with a name that codes western who had grammatical idiosyncrasies common to ESL Asians.

1) Why not?

2) You'd likely want a relatively common-sounding name, I suppose. Chinese names are relatively common-sounding.

A server with ssh generally implies there is a shell, and cli tools, and an administration model that involves humans connecting to servers to manually update them in place like pets.

Having a full workstation-optimized distro like ubuntu or debian with hundreds of packages constantly shifting and updating as a critical production server is wildly high risk in terms of both reliability and security.

I understand this is how most sysadmins were taught but it is a 70s unix mainframe mindset from a time before security was a thing and everyone on the internet was a good actor.

Only a workstation or dev server should have tools for humans like ssh installed.

Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust.

No production systems I launch today even have xz installed, or a package manager, or a shell.

Pay.

I mean, the idea of commitment -> privilege isn’t bad. But it’s got nothing to offer in the story of xz.

Why? My guess is that this new culture is promoted to extract free labor from hobby developers. Take for example, Github. It has a bot to close stale issues after a given interval of inactivity. But what if the issue was valid and unresolved? What's the problem in leaving it open for someone interested to take it up? But what GH wants to promote is a culture where open issues are considered as bad performance on the maintainer's side. It indirectly prompts the maintainers to take open issues too seriously.

I somewhat get it.

I became the central spot for this particular product. I get occasional emails from people saying the did this same thing, but on a smaller scale. I have scale, so I can do things much bigger than anyone else.

I have a million people using this, vs a startup who has ~100.

(Btw, I'm barely profitable, I was selling to frugal people...)

Having temp workers come and go into all kinds of various open source projects.. does that help? :)

Why not play into the strengths of OSS instead of trying to replicate frictionous trust models from elsewhere?

Like, listen to ourselves. We are admitting here that open source software is not feasible to audit? Like did we just accept that an adversary can basically smuggle exploit payloads in plain sight?

Can we at least try to simplify and reduce the messiness of checked in generated code, irreproducible builds, and magical binary blobs, conditional compilation, “ifuncs”? Wtf? Granted I’m not a domain expert in this area, but I’m horrified by all the complexity for what I understand to be a compression library.

I would much rather play whack-a-mole with obscurity, which as a side effect is great for software quality overall.

The best case scenario for a project that loses its BDFL(s) is to hold on maybe as decently as Apple has under Tim Cook. They haven’t done anything truly innovative since Jobs died in my opinion — they just figured out more ways to apply the combination of a multitouch display, compact computer, and sensors; but they’ve iterated well on everything they produce and have held on to their lead in terms of hardware, user experience, and ecosystem cohesion.

In short: someone very competent has to care a lot about the project for the right reasons, and continue driving it forward. If the old guard is rotating out, someone has to step up.

If no one steps up, then leeches of various types will attach themselves to the project and simultaneously milk it for value and kill it.

Such things can always be automatically discovered.

https://github.com/eliemoutran/KnockIt

Any effort spent on setups like this is likely better spent removing the need for having ssh at all by moving to immutable appliance distros be they specialized like homeassistant or general purpose like TalosOS.

That said, just make sure you aren't using the backdoored versions. I don't see how you can reasonably skip this step.

Look I get it, it's a great line, it instantly weeds out anyone who can't code, it even filters out those who can code but don't actually care enough about your project. If you choose that route though, you better be prepared for the day when the fabled PR finally arrives in your inbox. You told them to do it themselves if they wanted it so bad, they went and did it. Even though you know your code base and could probably have implemented it 10 times faster than they did, they labored over it to spare you the trouble instead of begging for the feature in your issue tracker. You asked to see the code, you got the code. Now the buck stops with you.

Seriously, if you say this, be ready to engage with whomever does show up with a patch. Unless they send you literal irredeemable garbage, the least they are entitled to is your sincere peer review. Nothing's worse than putting in effort into learning somebody else's project and writing the patch, only to not be given the time of day, or worse. I'd rather have a Linus Torvalds type take a big shit on the patch in public than be ignored like that. Someone who does that is at least treating you like a peer.

This has been reported and discussed from the start, even in the initial report. Also it's not a Linux specific patch or issue, rather a patch used by some Linux distros for tighter systemd integration.

However that would've likely done little to impede this attack if it is backed by organized crime or by a state as is being speculated. It is trivially easy for those types of actors to simply use a stolen identity or create an entirely plausible one out of thin air.

I don't think this is really a structural problem requiring these kind of sweeping changes; it's just an occasional rare incident.

The SaaS/cloud/etc. companies themselves should fund the projects they depend on. They actually know what those are and don't have to force their own monetization/growth models onto the projects.