Timeline of the xz open source attack
research.swtch.com
research.swtch.com
One thing that could be added, for the sake of completeness: in the part "Attack begins", toward the end, when they are pushing for updating xz in the major distros, Ubuntu and Debian are mentioned but not Fedora.
Looks like the social engineering/pressuring for Fedora started at least weeks before 2024 March 04, according to a comment by @rwmj on HN [1]. I also found this thread on Fedora's devel list [2], but didn't dig too much.
[1] https://news.ycombinator.com/item?id=39866275
[2] https://lists.fedoraproject.org/archives/list/devel@lists.fe...
He notes that while he was unsatisfied with some of the changes Tan introduced, Tan was nonetheless extremely helpful.
That's one way to put it
https://lists.fedoraproject.org/archives/list/devel@lists.fe... (https://archive.ph/diGNB)
This is the rough sequence of events in Fedora:
https://lists.fedoraproject.org/archives/list/devel@lists.fe... (https://archive.ph/e0SdX)
Its time to stop accepting that the way we've done this in the past is the way we will continue doing it ad infinitum.
Sounds like it would be impossible but maybe my thinking is just enclosed and not free.
It's worse than that, these small C files are required to fail unless the target build environment meets some specific requirements.
Configure scrips tend to be a very slow part of projects, so a new tool that would support parallel compile/run of these checks would also have a considerable speedup. In addition to being more reviewable, verifiable and less error prone.
I've been porting and compiling Unix programs since 5 years before Linux. There was a lot of work needed to port between Unix variants, or even worse, non-Unix variants. So I can see the problem that the autotools solve.
But I never spent much time learning the details on how the autotools work. When './configure' has some problem, it is a very steep uphill battle to get any traction on the issue.
So I'd be more in the camp of looking for a different solution than autotools.
So a first pass should probably be to eliminate as much checks as possible. But there will almost always be a need for a couple of build time checks/switches - should be supported by some kind of tooling.
https://cmake.org/cmake/help/latest/manual/cmake-modules.7.h...
So it seems considerably better than configure scripts. Probably also better than my idea :p I don't know how easy it would be for an adversary to do something "underhanded" though, or how easy that would be to spot such.
There might have been a time where it worked, but people started to use plugins for all kinds of reasons, quite good ones in most cases. Findbugs, code coverage, source code generation,...
Today, a maven project without plugins is rare. Maven brought us 95%, but there is a long tail left to cover.
For the purpose of this conversation we mostly just care about the use case of someone grabbing the code and wanting to use it in their own project. For this use case, dev tools like findbugs and code coverage can be ignored, so it would suffice to have a version of the build system with plugins completely disabled.
Code generation is the thornier one, and we can at least be more principled about it than "run some arbitrary code", and at least it should be trivial to say "this codegen process gets absolutely no I/O access whatsoever; you're a dumb text pipeline". But at the end of the day, we have to Just Say No to things like this. Even if it makes the codebase grodier to check in generated code, if I can't inspect and audit the source code, that's a problem, and arbitrary build-time codegen prevents that. Some trade-offs are worth making.
I'm not a fan of generated code. It tends to cause misery, being in a no mans land between code and not-code. But it is usefull sometimes, e.g rust generating an API from the opengl XML specs.
Sandboxing seems the least worst option, but it will still be uninspected half code that one day ends up in production.
The code was only provided in a roundabout way that was deliberately done to evade manual inspection, so that's not a failure of checking in generated code, that's a failure of actually building a binary from the artifacts that we expect it to be built from. Suffice to say, cutting out the Turing-complete crap from our build systems is only one of many things that we need to fix.
I'm not sure how well isolated the containers are (probably not very – I think GitHub gives access to the Docker socket) and you'd have to make sure they don't share secret tokens etc., but at least it might make things simpler to audit, and isolation could be improved in the future.
There is a sizable movement in the Rust ecosystem to move all build-time scripts and procedural macros to (be combined to) WASM. This allows you to write turing-complete performant code to cover all use-cases people can reasonably think of, while also allowing trivially easy sandboxing.
It's not perfect, for example some build scripts download content from the internet, which can be abused for information extraction. And code generation scripts could generate different code depending on where it thinks it's running. But it's a lot better than the unsandboxed code execution that's present in most current build systems, without introducing the constraints of a pure config file.
I’d argue that Rust’s biggest advantage here is that build scripts and procedural macros are written in Rust themselves, making them easier to read and thus harder to hide things in than autotools’ m4-to-shell gunk.
But that means it’s important to build those things from source! Part of the movement you cite consists of watt, which ships proc macros as precompiled WebAssembly binaries mainly to save build time. But if watt were more widely adopted, it would actually make an attacker’s life much easier since they could backdoor the precompiled binary. Historically I’ve been sympathetic to dtolnay’s various attempts to use precompiled binaries this way (not just watt but also the serde thing that caused a hullabaloo); I hate slow builds. But after the xz backdoor I think this is untenable.
If everything is done carefully enough with reproducible builds, I think using a binary whose hash can be checked shouldn't be a great extension of trust.
You could have multiple independent autobuilders verifying that particular source does indeed generate a binary with the claimed hash.
But determined actors with access are always going to try to thwart these schemes, so verification and testing is going to need to step up.
if you trust that they're giving you the correct hash, but not the correct binary, then you're not thinking clearly.
To put it another way: if you don't have reproducible builds and you're trying to replicate an upstream artifact then it's very hard to tell what the cause is. It might just be some ephemeral state, a race, or some machine aspect that caused a few fairly meaningless differences. But as soon as you have a reproducible build, then failure to reproduce instantly marks upstream as being suspect.
It's also useful when you're trying to do things like tweak a build - you can ensure you're importing upstream correctly by checking the hash with what you're making, and then even if your downstream version isn't a binary dependent on upstream (e.g. optimized differently, or with some extra plugins somewhere), you can be sure that changes between what you're building and upstream are intentional and not spurious.
It's clearly not a silver bullet, sure. But it's not entirely useless either, and it probably could help as part of a larger ecosystem shift; especially if conventional tooling created and published these by default such that bad actors trying to hide build processes actually stick out more glaringly.
Debian's response is looking to be mandating the source must come from the git repository, not a tar ball. And it will be done using a script. And the script must produce the same output every time, ie be reproducible. Currently Debian's idea of "reproducible" means it reflects the source Debian distributes. This change means it will reproducibly reflect the upstream sources. That doesn't mean it will be the same - just that it's derived in a reproducible way.
As for trusting the person who gave you the hash: that's not what it hinges on. It hinges on the majority installing a binary from a distro. That binary can be produced reproducibly, by an automated build system. Even now, Debian's binaries aren't created just once by Debian. There are Debian builders all around the planet creating the binary, and verifying it matches what's being distributed.
Thus the odds of the binary you are running not being derived from the source are vanishingly low.
I see what you’re saying, but I don’t buy that reproducible builds actually solve anything, especially long term. As this whole xz thing has shown us, lots of things fly under the radar if the circumstances are right. This kind of thing will absolutely happen again. In the future it may not even require someone usurping an existing library, it could be a useful library created entirely by the hacker for the express purpose of infiltration a decade later.
Reproducible builds are a placebo. You must still assume there are no bad actors anywhere in the supply chain, or that none are capable of hiding anything that can be reproducibly built, and we can no longer afford to make that assumption.
A reproducible vulnerability is still a vulnerability.
A window that can be smashed in is still a vulnerability, so there is no value in people locking their front doors.
You're right. They don't solve anything, if you restrict "anything" to mean stops exploits entering the system. What reproducible builds do is increase visibility.
They do that by ensuring everybody gets the same build, and by ensuring everybody has the source used to do the build, and by providing a reliable link in the audit trail that leads to when the vulnerability was created and who did it. They make it possible to automatically verify all that happened without anybody having to lift a finger, and it's done in a way that can't be subverted as Jia Tan did to the xz utils source.
Ensuring everyone has the same build means you can't just target a few people, everyone is going to get your code. That in turn means there will be many eyes like Andres looking at your work. Once one of them notice something, one of them is likely to trace it back to the identity that added it. Thus all these things combine to have one overall effect - they ensure the problem and it's cause are visible to many eyes. Those eyes can cooperate and combine their efforts to track down the issue, because they are all are absolutely guaranteed they are looking at the same thing.
If you don't think increasing visibility is a powerful technique for addressing problems in software then you have a lot to learn about software engineering. There are software engineers out their who have devoted their entire professional lives to increasing visibility. Without them creating things that merely report and preserve information like telemetry and logging systems software, would be nowhere as reliable as it is now. Nonetheless, if the point you are making is when we shine a little sunlight into a dark corner, the sunlight itself does nothing to whatever we find there, you're right. Fixing whatever the sunlight / log / telemetry revealed is a separate problem.
You aren't correct in thinking visibility doesn't reduce these sorts of attacks. By making the attack visible to more people, it increases the chance it will be notice on any given day, thus reducing it's lifetime and making it less useful. Worse, if the vulnerability was maliciously added the best outcome for the identity is what happened here - the identity was burned, all its contributions where also burned and so a few years of work went down the drain. At worst someone is going to end up in jail, or dead if they live in authoritarian state. The effect is not too different from a adding surveillance cameras in self-checkouts. Their mere presence makes honesty look like a much better policy.
If you trust the people giving you the hash of the binary, just get the binary from them, too.
Reproducible builds and hashes thereof aid in transparency and detecting when that transparency breaks down. Of course, it doesn't mean hackers can't hide malicious code in plain sight, but at least it makes it slightly harder to hide between the cracks as happened here.
The days of 3rd party libraries simply being trusted because they're open source are slowly coming to an end. the problem is not unreproducible builds, the problem is that we implicitly trust source code found online.
Projects relying on 100s of 3rd party libraries are a major problem, and no one seems to care. They like it when their Rust build lists all the libraries that they didn't have to write as it compiles them. They like it when they can include an _extremely_ small Node library which itself relies on 4 other _extremely_ small Node libraries, which each rely on another 4 _extremely_ small Node libraries, until you have a node_modules directory with 30,000 packages in it.
We don't know how to write software any more. We know how to sew bits of things together into a loincloth, and we get mad when the loincloth makes us itch. Of course it does, you didn't LOOK at anything you are relying on. You didn't stop to think, even for a moment, that maybe writing the thing instead of trusting someone else with it was even an option.
As we continue doing this, relying on 3rd parties to write the heavy lifting code, we lose the skill to write such code ourselves. We are transferring the skills that we need into the hands of people that we can't trust, when viewed from a high level.
We need to get back to small applications, which we know because we wrote them in their entirety, and that someone else can trust because the code for the thing is 2500 lines long and has extremely few, if any, dependencies.
We need to get away from software which imports 30,000 (or even 100) third party libraries with implicit trust because it's open source.
"All bugs are shallow with enough eyes" requires that people use their eyes.
Or in other words, the point is not for everyone to verify the build produces the expected binary since that would indeed make the published binaries pointless. Instead, most people trust the published hash because it can be independently verified and anyone can call out the publisher of the binary if it doesn't match.
I’m not sure I follow. The bar for passing review and audits is “people can understand it”, at least to the point of preventing smuggling in a custom public keys and other shenanigans – as opposed to “all valid programs in the language”. While I agree Turing-completeness opens up undeniable complexity, it’s the best we got. And there are huge variations in obscurity across languages and tooling.
I would like to see this on the language level, not just on the OS level.
Say I have a directory with a virtualenv and some code that needs some packages from PyPI. I would very much like to sandbox anything that runs in this virtualenv to just disk access inside that directory, and network access only to specifically whitelisted URLs. As a user I should only need to add "sandbox = True" to the pyproject.toml file, and optionally "network_whitelist = [...]".
From my cursory looking around, I believe Cloudflare sandbox utils, which are convenience wrappers around systemd Seccomp, might be the best starting point.
Edit: or just use Firejail, interesting...
You mention sandboxing on the language level, but I don't think it is the way. Apparently sandboxing within Python itself is a particularly nasty rabbit hole that is ultimately unfruitful because of Python's introspection capabilities. You will find many dire warnings on that path.
> 99.9% of all projects have the capability to be built with trivial declarative rulesets.
Only if you forbid bootstrapping, which all projects ultimately rely on at some point in their supply chain.
We could write an entire PhD thesis on the number of dire technical failings that would need to be addressed to stop this attack, so while this alone wouldn't have stopped it, it would have required the actor to come up with another vector of code injection which would have been easier to find.
> Only if you forbid bootstrapping
Codebases that bootstrap are the 0.1%. Those can be built via `bash build.sh` rather than deceptively hiding a Turing-complete environment behind a declarative one. Even if you need to have these in your trusted computing base somewhere, we can focus auditing resources there, especially once we've reduced the amount of auditing that we need to do on the other 99.9% of codebases now that we've systematically limited the build-time shenanigans that they can get up to.
At worst, you're actually creating more holes. The reason autoconf/automake exist and M4 scripts are innocuous in the first place is because the build system uses an underpowered language and developers have to turn to code generation to get around it.
If you kneecap the build system's language you're not solving problems. You're creating them.
> it would have required the actor to come up with another vector of code injection which would have been easier to find.
If make was standardized and could programmatically determine the environment its run under and write full programs then the attack vector wouldn't exist in the first place.
> Codebases that bootstrap are the 0.1%.
We have different experiences, because ime it's close to 100% especially when you include transitive dependencies. When you care about supply chain security you care about being able to bootstrap from sources for your code and all your dependencies, and it's almost guaranteed that one of your dependencies needs to be bootstrapped.
Reading the timeline, the root cause is pure social engineering. The technical details could be swapped out with anything. Sure there are aspects unique to xz that were exploited such as the test directory full of binaries, but that's just because they happened to be the easiest target to implement their changes in an obfuscated way, that misses the point - once an attacker has gained maintainer-ship you are basically screwed - because they will find a way to insert something, somehow, eventually, no matter how many easy targets for obfuscation are removed.
Is the real problem here in handing over substantial trust to an anonymous contributor? If a person has more to lose than maintainership then would this have happened?
That someone can weave their way into a project under a pseudonym and eventually gain maintainership without ever risking their real reputation seems to set up quite a risk free target for attackers.
It could also be called an industry problem, where we rely on other's code without running proper checks. This seems to be an emerging realization, with services like socket.dev starting to emerge.
Unless there's some practical way of comprehensively solving The Real Problem Here, it makes a lot of sense to consider all reasonable mitigations, technical, social or otherwise.
> If a person has more to lose than maintainership then would this have happened?
I guess that's one possible mitigation, but what exactly would that look like in practice? Good luck getting open source contributors to accept any kind of liability. Except for the JiaTans of the world, who will be the first to accept it since they will have already planned to be untouchable.
It's not necessary to accept liability, that's waived in all FOSS licenses. What I'm suggesting would only risk reputation of non-malicious contributors, and from what I've seen, most of the major FOSS contributors and maintainers freely use their real identity or associate it with their pseudonym anyway, since that attribution comes with real life perks.
Disallowing anonymous pseudonyms would raise the bar quite a bit and require more effort from attackers to construct or steal plausible looking online identities for each attack, especially when they need to hold up for a long time as with this attack.
It's heartblead all over again (though those were honest mistakes, not an intentional attack
And all code that gets linked into security critical applications/libraries needs to be covered by under some sort of security focused code review.
So no patching the compression code that openSSL links to with random junk distribution maintainers.
Who will pay for all the rewriting you want done? Or even just for the new frameworks that are "scrutable"? How do we guarantee that the result is not inscrutable to you or others?
There is so much knee-jerking in this xz debacle.
(And I say this / ask these questions with no love for autoconf/m4/sh.)
Being able to send xz patches to the Linux kernel would have been a nice point of leverage for Jia Tan's future work. We're not at trusting trust [1] levels yet, but it would be one step closer.
Build infrastructure should be minimal, standardized, and not subject to endless special, undocumented fragility.
cmake, just, conan, meson, and bazel (and forks) exist.
But I've still yet to see a proper build system that does feature detection in parallel and concurrently, or supports live, incremental, continuous, cached rebuilding.
We should be moving from No Obvious Bugs to Obviously No Bugs (Tony Hoare).
That’s the beauty of FOSS: if you don’t like what’s written, you can write your own replacement.
How many security issues spring from outdated packages vs packages updated too hastily?
A newly-introduced security issue tends to have very limited exploitability, because it's valuable, not-yet well understood, and public exploits are yet to be developed.
Compare to that a similar vulnerability in an older package: chances are that everything about it has been learned and is publicly known. Exploits have become a commodity and are now part of every offensive security distro on the planet. If you run that vulnerable version, there's a real risk that a non-targeted campaign will randomly bite you.
Except in the scenario that is this exact case: Supply chain attacks that are developed with the exploit in mind.
For an old vulnerability, however, any unpatched system is a target. So the individual risk for the average unpatched system is still orders of magnitude higher than in the former scenario.
Another example of this was log4j: if you were still using the old 1.x log4j versions, you wouldn't have been vulnerable to the log4shell vulnerability, since it was introduced early in the 2.x series. The old 1.x log4j versions had other known vulnerabilities, but only if you were using less common appenders or an uncommon server mode or a built-in GUI log viewer (!); the most common use of log4j (logging into a local file) was not exposed to any of these, and in fact, you could remove the vulnerable classes and still have a functional log4j setup (see for instance https://www.petefreitag.com/blog/log4j-1x-mitigation/ which I just found on a quick web search).
Did log4shell (and a later vulnerability which could only be exploited if you were using Java 9 or later, because it depended on a new method which was introduced on Java 9) lead people to question whether always being on the "latest and greatest" was a good thing? No, AFAIK the opposite happened: people started to push even harder to keep everything on the latest release, "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".
Lol, this exact thing happened at my last gig. When I first learned of the vulnerability I panicked, until I found out we were so outdated it didn't affect us. We had a sad laugh about it.
> "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".
I think there is some truth to this motivation though - if you are on an ancient 1.X version and have to jump a major version of two, that almost always causes pain depending on how critical the service or library is. I don't pretend to know the right answer but I always tend to wait several versions before upgrading so any vulnerabilities or fixes can come by the time I get to the upgrade.
Isn't this still true, generally speaking? How many servers are daily compromised due to out of date, exploitable versions of software versus XZ-like sophisticated attack?
Now we're in a position where most projects update so quickly, that you don't really have a choice. If you need to update one component, there's a good chance it will require that many more as dependencies in your project will require an update to be compatible.
The industry as a whole sacrificed stability and some aspects of security for faster advancements and features. Overall I'd say the net benefit is positive, but its times like these that remind us that perhaps we need to slow things down just a little and do a bit of a course correction to bring things into a better balance.
Everyone being on different versions results in something like a moving target
I highly encourage people to try freeBSD sometime. Give ports a try (although the modern sense is that poudrie is better even if you want custom-built packages). See how nicely everything works. All the system options you need go into rc.conf (almost uniformly). Everything is documented and you can basically operate the system out of the FreeBSD Handbook documentation (it's not at all comparable to the "how to use a window or a menu" level intro stuff the linux provides). You can't do that when everything is furiously churning every release. everything just works, everything is just documented, it's an experience when you're coming from linux.
and that forum post from 2007 on how to tweak a service script is probably still valid, because BSD hasn't had 3 different init systems over that timespan etc.
just like "engineering is knowing how to build a bridge that barely doesn't fall over", engineering here is knowing what not to churn, and fitting your own work and functionality extensions into the existing patterns etc. like it doesn't have to be even "don't make a bigger change than you have to", you just have to present a stable userland and stable kernel interface and stable init/services interface. the fact that linux doesn't present a stable kernel interface is actually fairly sketchy/poor engineering, it doesn't have to be that way, a large subset of kernel interfaces probably should be stable.
I still think this is the way to go, but it does move some of the problem around in that you have to pick good cleave lines such that people aren’t forced to pull 5 libraries in for 5 features they need. You want one software domain to use two or three and another to use an overlapping set of three.
Supply chain attacks are a real threat too, but being conservative about upgrading to improve your security posture is like saying "I'm worried someone with the key to my house might rob me, so I'm going to remove the lock from my door."
If being rude and pushy doesn’t work, the next round will be kind and helpful. Don’t read too much into the cultural techniques used, because the cultural techniques will mirror the culture at the time.
If they want things done quickly they can do it themselves
I mean they kind of did. And that was the problem.
I think we need to start paying more of these open source maintainers and have some staff/volunteers that can help them manage their git hub issue volume.
A user reported a safety issue, the maintainer said it was safe. Then it was proven that it was in fact unsafe, and the maintainer justified it with performance. Then a PR was filed which was safe and did not regress performance, and the maintainer rejected it with "this patch is boring"
The behavior of both sides was deeply unacceptable. If someone identifies a legitimate issue and files a PR to fix it, don't insult them by calling the patch "boring" and don't reject it solely on that basis.
There's a certain kind of talented person who is all too conscious of their abilities and is arrogant, irascible, and demanding as a result. Linus Torvalds, Steve Jobs, Casey Muratori come to mind. Much as we might want these characters to be kinder, their irascibility is inseparable from their more admirable qualities.
Sometimes good things, even the best things, are made by difficult people, and we would lose a lot by making a community that alienates them.
[0] clip about the Nvidia "incident" https://youtu.be/wvQ0N56pW74
[0] Funnily enough, just a week or two ago, I fixed an issue in an OS project that we introduced at work. It was an easy frontend fix even for someone like me who doesn't do frontend and barely knows how to spell Vue. And more importantly, in the issue description somebody already wrote exactly what causes the bug and what would need to change - the only thing left was finding the place where to make the (one-line) change. Somehow that issue had been open for 2 years but nobody of the several people who complained (nor the maintainer) had bothered to fix it. After I made a PR, it was merged within a day.
I did an engineering/program manager role for 8 years and people pretty much always did what I asked if I showed up at their desk or bothered their boss.
"Squeaky wheel gets the grease?"
But I too like to think that I prioritize my children on merit rather than fuss level. For some reason they continue to cry despite me claiming I don't react to it.
We've complained about bots in social media for a long time, but how many people in open source discussions are shady manipulative entities?
Given the current situation, I'm slightly worried about Fedora's planned move to zlib-ng instead of zlib in the next release (https://fedoraproject.org/wiki/Changes/ZlibNGTransition).
I wonder if it would be possible to crowdsource FOSS mailing list moderation.
For lesser/individual maintainers there is no way to survive this kind of mob attack. Corporate maintainers may be able to manage as it could be considered just as paid job and there are worse ways to make money.
First send a message to the mailing list as "Alice" providing a detailed bug report for a real bug, and a flawless patch to fix it.
Then you reply to the mailing list as "Bob" agreeing that it's a bug, thanking "Alice" for the patch and the time she spent producing such a detailed bug report, then explaining that unfortunately it won't be merged any time soon, then apologising and saying you know how frustrating it must be for Alice.
Your two characters have been model citizens: Alice has contributed a good quality bug report, and code. Bob has helped the project by confirming bug reports, and has never been rude or critical - merely straightforward, realistic and a bit overly polite.
Play is just preparing for the same game but when stakes are higher?
How to identify and defuse: https://www.pon.harvard.edu/daily/batna/the-good-cop-bad-cop...
We live in an adversarial environment, time to stop playing naively nice. Ideally it isn't the maintainer that has to do all this work.
Messages like Jigar’s are kind of par for the course.
Yeah, same. We should be much more willing to kick jerks out of our work spaces. The work is hard enough as it is without also being shit on while you do it.
Simple ban and get the fuck out. Too often I've dealt with people trying to rationalize it as much as "o its just cultural, they don't understand". No, get the fuck out.
But hey I'm a NYer and telling people to fuck off is a past time.
Being mean is stressful and stops your brain from working properly. If someone doesn't allow you to be abusive, then they are not a mark. Predators look for prey that falls into certain patterns.
I think this could be a genuine use of an AI: to go through all of the shit, and have it summarized in a fashion that the user wants: distant and objective, friendly, etc. It could provide an assessment on the general tone, aggregate the differently phrased requests, many things like that.
Crowdsourcing would works best with the reddit / hacker news model I feel, where discussion happens in tree styled threads, and users can react to messages in ways that are not text, but something meta, like a vote or a reaction indicating tone.
Both of these have significant downsides, but significant upsides too. People pick the mailing list in a similar way.
Trying to do something about it was called censorship.
I myself do not think that this is the actual problem. I think the actual problem is that many FOSS communities have fostered an idea that cracking down on certain types of behavior is censorship. Of course they might all agree to ban somebody who is saying certain well known explicit words or being really really aggressive, but there's a lot of behavior that I would consider bannable but they do not.
I'm trying to avoid mentioning specific cases because I don't wanna either reopen old wounds or cause another similar dustup. I can say that it reminds me a lot of the really old post about the 5 geek social fallacies https://plausiblydeniable.com/five-geek-social-fallacies/ In particular social fallacy #1
Obviously we're not talking about geek culture here (in general anyways). After rereading it, I saw something else that we actually do a see a lot even today.
This in particular has happened on many a mailing list, including the ones I'm not specifically mentioning
>> <SNIP> when nothing smacking of social selectiveness can be discussed in public, people inevitably begin to organize activities in secret. </SNIP>
Well-known annoyances get tolerated for years, and the people who do the work start moving their discussions to bug trackers, irc, or in personal emails to avoid interacting with them.
In another thought, I hope the nukes aren’t on a rolling Debian distro.
As a hobbyist developer/maintainer of open source projects, I strive to remember that this is my gift to the world, and it comes with no strings attached. If people have any expectations about the software, it’s for them to manage; if they depend on it somehow, it’s their responsibility to ensure timely resolution of issues. None of this translates to obligations on my part, unless I explicitly make promises.
I empathize with Lasse having been slowed down by mental issues. I have, too. And we need to take good care of ourselves, and proactively prevent the burden of maintainership from exacerbating those issues.
[0]: https://gist.github.com/g1eny0ung/9e7d4d0f72547a8d156452e76f...
If there is a law that would entitle a user to more, it is a bug in legislation that needs urgent fixing.
Anyone and everyone in the OSS world should be concerned about this too. You have nation state level actors out there with massive amounts of information on you. How much information have you leaked to data brokers? These groups will know how much debt you're in. The status of your relationships. Your health conditions and medications? It would not take much on their part to make your life worse and increase your stress levels. Just imagine things like fake calls from your bank saying that debt of yours has been put in collections.
(1) xz and the lib are widely used in the wild including linux kernel, systemd, openSSH; (2) single maintainer, low rate of maintenance; (3) the original maintainer has other problems in his life distracting them from paying closer attention to the project.
I am wondering how many other OSS projects look similar and can be targeted in similar ways?
Controlling someone's lights is probably less important than Debian's build fleet but it's a scary proposition for the impacted individual who happens to use one of those long tail home assistant integrations or whatever.
We're in a tech slowdown right now. There are people who got used to a certain lifestyle who now have "seeking work" on their LinkedIn profiles, and who have property taxes in arrears that are listed in county newspapers-of-record. If you're an intelligence operative in the Silicon Valley area, these guys should be easy pickings. An envelope full of cash to make some financial problems go away in exchange for a few commits on the FOSS projects they contribute to or maintain.
Are you married? Have a house? Pets? Children? Sick parent? Gay? Trans? Mental health issues? Disabled? All of this can be used against you. Be careful where and how you share stuff like this. I know it's not "cool" to be mysterious online anymore, but it creates a much larger attack surface. People can still engage with groups around these things, but better to do it with various personas than to have one trackable identity with everything attached to it.
Put a link saying "Hey this guy forked my project, I won't maintain it anymore, he may add malware, review and use at your own risk"
Yes, if you can't find a successor you trust then let someone fork the project and build trust from 0 rather than transferring trust others' trust in you by handing over the project. This doesn't just apply to security concerns btw. - plenty of other ways in which a new maintainer might end up making the project worse (intentionally or through incompetence) compared to it not receiving any updates.
It's time for more of the big vendors who use these projects in their offerings to step up and give people running these small projects more resources and structure. $20k to have maintainers for each project actually meet twice a year at a conference is chump change for the biggest vendors, especially when compared against the cost of the audits they'll now be doing on everything Jia Tan and Co. touched.
Now maybe one could build a career out of OSS maintainerships, with work/time funded by lots of donations much smaller than a salary but amounting to a salary.
Furthermore, these organizations are in a place to put experienced, trustworthy contributors on projects that need maintainers if need be. If Lasse had been able to go to, idk, the Linux Foundation and say, "Listen, I'm getting burnt out, got anyone?" and they said "Sure, we've got this contributor with an established record who would love to help maintain your project", none of this is happening right now.
He was also turned on by multiple people who he considered close friends. In the book it did not seem that he had considered that it might not be a "them" problem.
*my details may be off here, I read it some time ago
So much for the folks advocating for binary (driver) blobs in OSS t support otherwise unsupported hardware.
It's either in source form and reproducable or it's not there.
Wanna know how I know you haven't read into the discussion much?
There are a whole lot of binary test cases in software. Especially when you're dealing with things like file formats and test cases that should specifically fail on bad data of particular types.
That's not how I read GP's point. If even binary blobs in test cases are a place where backdoors are, now as a matter of fact, hidden then, certainly, among the folks advocating for binary drivers in FOSS, there are some who are already --or planning to-- add backdoors there.
Binary blobs are all terrible, terrible, terrible ideas.
Builds should be 100% reproducible from source, bit for bit. At this point it's not open up for discussion anymore.
Instead of obscure constants, you use known constants, or at least simple methods to derive your constants.
You can do the same thing to come up with your test cases. Bad zip? Construct a good zip of 10 files, each containing the first 10,000 prime numbers. Then corrupt the zip by seeking to position (100/pi) and write a thousand zeroes there.
Bad JPEG? Use Imagemagick to render the first 1000 prime numbers as text into a JPEG file, then apply a simple nothing-up-my-sleeve corruption operation.
There are still cases where this approach isn't going to work: that new icon, helpfully proposed by a contributor, meant to be used in production, might contain malicious code, steganographically embedded. I think there's little you can do to prevent that.
[1]: https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number
Forbidding binary data in source repositories is neither practical nor solving the issue.
Binary code can be hidden steganograhically in e. g. image files or text files.
That's the root cause. At some point a corp/gov consortium needs to adopt key projects and hire out the maintainers and give them real power and flexibility, perhaps similar to the way a nation might nationalize key infrastructure.
But this is against the core ethos at some level. Freedom and safety can be antagonistic.
Personally I wouldnt trust a govt to not backdoor everything.
I'm implying they are richer.
This is the second time I've read this "observation", but this observation is just wrong? Jigar's email is "${name}${number}@${host}", yes, but Dennis's is just "${name}@${host}" — there's not a suffixed number. (There's a 3, but it's just a "leetcode" substitution for the E, i.e., it's semantically a letter.)
(They could still both be sockpuppets, of course. This doesn't prove or disprove anything. But … let's check our facts?)
somewhat ironic, but I'd say effective.
At a certain point, companies need to step up and provide funding or engineering work or they should just keep expecting to get owned.
That doesn't mean we shouldn't try to help burnt out committers, but the problem seems very hard. As lcamtuf also says, many things don't need maintenance, and just paying people doesn't address what happens when they just don't want to do it anymore. In an alternate universe with different leadership, an organization like the FSF might use donated funds to pay a maintenance staff and an open-source maintainer might be able to lean on them. Of course, that still doesn't address the problem of Jia Tan getting a job with this organization.
https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-ba... https://lcamtuf.substack.com/p/oss-backdoors-the-allure-of-t...
I suspect some variant of this will grow so that some companies, MS/GitHub for example, audit large body of code and vet it for everyone else.
I admit, the op has a "professional/polished vibe" to me as well, but we seem to know very little except for what work time/zones were preferred by the possibly collective/possibly singular human(s) behind the Jia Tan identity. Does anyone have slick linguistic tools to assess if the writer is a single author? Maybe an opportunity to show off.. It's sort of how they caught Ted Kaczynski.
It also absolutely makes sense to think of all the state actors (I agree including as you say the US/UK) as part of the ongoing threat model. If the KGB/Ministry of State Security/NSA/MI6 were not doing this before then they surely might in the future. Maybe with more gusto/funding now! They all seem to have an "information dominance at all costs" mentality, at least as agency collectives, whatever individuals inside think.
Anyway, I've yet to hear of anything in the xz work beyond the ability of 1-3 skilled black hats with a lot of post-COVID time on their hands. The NSA ensuring the Intel ME/AMT could be turned off seems another level of sophistication entirely, for example { a "higher pinnacle" in your nice phrasing :-) }.
In terms of sheer numbers, my impression is that the vast majority of attacks blocked at almost every sophistication level are more "criminals" than "states". Admittedly, that may partly be states just acquiring ability to act surgically rather than launching big compromise initiatives like botnets (or otherwise states going undetected). I'm sure it's hard to know.
Maybe we're just in a sad era of increasing attack sophistication measured in "Kilo-Sophistic-meters"? The AntiVirus industry has been having a boom lately.
It's probably already been mentioned many times, but besides economic & geopolitical incentives, maybe the attacker(s) was a hard core `systemd` or IBM/RedHat hater or one of the people who supposedly issued death threats to Lennart Poettering now at Microsoft or even an open source hater burnt out or wanting to burn the world down like The Joker in Batman. In light of that, Russ' Setting The Stage Prelude could perhaps profitably add the introduction of that lib dependency into `systemd` and also various distros defaulting to `systemd`.
Anyway, premature conclusions are to be cautioned against over & over. That's all I was trying to do. { And I'm not claiming you were concluding anything - you seem pretty open-minded about it all. I was always just amplifying parent comments. I absolutely agree long games "feel" more State Actor - part of what I meant by "vibe", but to quote the detective in V For Vendetta - "it's just a feeling". To Old People, 2 years famously doesn't seem as long as to a 25- or 15-year old. ;-) It's actually short on the 5+ year Linux distro maintenance time scales. }
If I had ever received a request from a well-known figure with a longstanding reputation, who's appeared in-person at talks and has verifiable employment, I might've been more receptive. But all the requests I got were from random internet identities that easily could've been fabricated, and in any case had no previous reputation. "Jia Tan" and their other sockpuppets very likely are not their real identities.
In this scenario, Jia Tan would not be a suitable replacement, since they don't friggin' exist.
Yes, there's problems with this approach. When money gets involved, incentives can get distorted. It limits the pool of acceptable maintainers to those employable by a Big Company. But I think these are solvable problems, especially if there's a strong culture of maintainer independence. It provides a real improvement over the current situation of putting so much pressure on individuals doing good work for no benefit.
Is it a solution at all? Say Oracle offer Lasse Collin $200k for maintaining xz but he doesn't want to work for Oracle so he refuses, then what? Amazon offer Lasse $200k but they require fulltime work on other open source packages which he isn't experienced with or interested in, so he refuses, then what? Google employ someone else for $200k but they can't force Lasse Collin to hand over commit rights to xz to Google, or force him to work with a fulltime Google employee pestering with many constant changes trying to justify their job, and they can't force Debian to accept a new Google fork of xz, so then what? And NetFlix, Microsoft, Facebook, Uber, they can't all employ an xz maintainer, xz doesn't need that many people, but if they just employ 'open source maintainers' scattering their attention over all kinds of random projects they have no prior experience with, how would they catch this kind of subtle multi-pronged long-term attack on some low-attention, slow moving, project?
Google already employ a very capable security team who find issues in all kinds of projects and publicise them, they didn't find this one. Is it likely this attack could have made its way into ChromeOS and Android if it wasn't noticed now, or would Google have noticed it?
> "1) don't make everyone so mad that everyone on the Internet asks for you to be fired"
So it's a sinecure position, doing nothing is the safest thing?
> "and 2) name a suitable replacement when you're ready to move on"
How could Lasse Collin have named a more suitable replacement than someone who seemed technically capable, interested in the project, and motivated to work on some of the boring details like the build system and tests and didn't seem to be doing it for the hype of saying "I improved compression by 10%" for their resume? Are they needing to be skilled in hiring and recruitment now?
> but if they just employ 'open source maintainers' scattering their attention over all kinds of random projects they have no prior experience with
They would pay the people who are doing the work now. Under this hypothetical, one of the Big Companies would have hired Lasse as the xz maintainer, for example. His job responsibilities are to maintain xz as he had been doing, and identify a successor when he's ready to move on. Nothing else.
> So it's a sinecure position, doing nothing is the safest thing?
No. Not doing the maintenance tasks would make everyone mad, violating one of the two job responsibilities.
> How could Lasse Collin have named a more suitable replacement than someone who seemed technically capable, interested in the project, and motivated to work on [it]
Lasse would suggest Tan as a suitable replacement. Big Company's hiring pipeline would approach Tan and start the hiring process (in person interviews, tax docs, etc etc). At some point they would realize Tan isn't a real person and not hire him. Or, the adversary would have to put up "a body" behind the profile to keep up the act, which is a much higher bar to clear than what actually happened.
> "Lasse would suggest Tan as a suitable replacement. Big Company's hiring pipeline would approach Tan and start the hiring process (in person interviews, tax docs, etc etc). At some point they would realize Tan isn't a real person and not hire him"
What if they find that Tan is a real person but he doesn't want to work for Amazon or legally can't (This is before knowing of his commits being malicious, we're assuming he's a fake profile but he could be a real person being blackmailed)? Collin can't leave? Collin has to pick someone else out of a candidate pool of people he's never heard of? Same question if they find Tan isn't a real person - what then; is there an obligation to review all of Tan's historical commits? Just committing under a pseudonym or pen name isn't a crime, is it? Would the new maintainer be obliged to review all historic commits or audit the codebase or anything? Would Amazon want their money back from Lasse once it became clear that he had let a bad actor commit changes which opened a serious security hole during his tenure as maintainer?
> "No. Not doing the maintenance tasks would make everyone mad, violating one of the two job responsibilities."
What he was doing already was apparently ignoring issues for months and making people like Jigar Kumar annoyed. Which is fine for a volunteer thing. If "Jigar Kumar" is a sock-puppet, nobody knew that at the time of their posts; Lasse Collins' hypothetical employer wouldn't have known and would surely be on his case about paying him lots of money for maintenance while complaints are flowing on the project's public mailing list, right? Either they're paying him to do what he was doing before (which involved apparently ignoring work and making some people mad) or they're paying him to do more than he was doing before (which is not what you said).
It doesn't seem like it would work - but if it did work it doesn't seem like it would have helped against this attack?
Well, there's a few issues I'm trying to target. I'm trying to work backwards from "how do we stop bad actor Tan from getting maintainer access to the project?" Creating an identify-verified relationship (employment) is a good fit for that, I think. And it nicely solves some other related issues with the current volunteer maintainership model. Lasse may not have felt the strong pressure/health issues if he was being paid to do the work. Or, if he was feeling burnt out, he may have felt more comfortable passing the torch earlier if there was a clear framework to do so, backed by an entity that can do some of the heavy lifting of naming/validating a successor.
> What if they find that Tan is a real person but he doesn't want to work for Amazon or legally can't
I think this would be a fairly rare occurrence, but it's one I called out as a potential problem in my original post, yeah ("smaller pool of possible maintainers"). If there isn't a clear successor, I think the maintainer could inform the Big Company that they'd like to move on in the next year or two, and Big Company could maybe find an internal engineer who wants to take over the role. Or maybe this more formal sponsored-maintainership arrangement would create incentives for outside contributors to aim for those positions, so there's more often someone waiting to take over (and then be verified by Big Company's hiring process).
> is there an obligation [for the maintainer] to review all of Tan's historical commits? Would the new maintainer be obliged to review all historic commits or audit the codebase or anything? Would Amazon [fire] Lasse once it became clear that he had let a bad actor commit changes which opened a serious security hole during his tenure as maintainer?
(I tweaked your questions a tiny bit to rephrase them as I interpreted them. I think the spirit of your questions was kept, I apologize if not.) If these tasks fall under the "don't make everyone mad" job responsibility, then yes. If not, then no, to all of these. There are no obligations other than the two I mentioned: don't piss off the community and help name a successor. It's up to the project's community to decide if the maintainer is not meeting their obligations, not the sponsoring Big Company.
> What he was doing already was apparently ignoring issues for months and making people like Jigar Kumar annoyed.
I'm not sure. It seems like Kumar was a bad actor. Was there actually a real maintenance issue? If so, maybe it could have been avoided in the first place by the sponsorship arrangement, like I mentioned at the top of this reply. Or, the community could raise the problem to Big Company, who can do the work of verifying that there is a problem and working with the maintainer to resolve it. Instead what happened here, which was for one burned out guy deciding to hand the keys over to some email address.
It would stop a sock puppet, but Jai Tan might be a real person, a real developer paid or blackmailed by a hostile group; Amazon might just have hired him and handed over maintainer access to him thinking it was above board, if a problem hadn't been found yet. I don't know where Jai Tan claimed to be from, but it's quite possible they would say "I don't have a passport", "I can't leave my family to travel to America for an in-person interview", "I'm not in good health to travel", "I don't speak English well enough for an in-person interview", "I live in a poor country without a functioning government and have no tax documents", or etc. etc. excuses which are quite plausible
> "Or, if he was feeling burnt out, he may have felt more comfortable passing the torch earlier if there was a clear framework to do so, backed by an entity that can do some of the heavy lifting of naming/validating a successor."
Your suggested $200k is equivalent to £160k GBP in the UK; look at this UK average salary list: https://uk.jobted.com/ no job comes close; not Managing Director, IT director, Finance Director, Aerospace engineer, DevOps engineer, neurosurgeon, nothing on the list is above £110k. Sure there are many people earning that much as a senior devops AI cloud security specialist in a fast paced London based fintech trading house, but the idea that someone would comfortably pass up a salary around the 98th percentile of incomes in the country for like 2 days a month of work because they're "feeling burnt out" is unthinkable. Anyone sensible would hold onto that until they pried it out of one's cold dead hands; American tech salaries are almost literally unbelievable. Even moreso if we consider a maintainer in a poorer country.
> "I tweaked your questions a tiny bit to rephrase them as I interpreted them. I think the spirit of your questions was kept, I apologize if not"
I started writing Tan, but then changed it. A lot of your reply is assuming that we know there were malicious patches and suspect Jigar Kumar was a bad actor and that the big company would be somewhat US friendly. But we can't plan to know all that for all situations like this. Some people will be speculating that the previous paid maintainer was complicit and all their work and merges are now suspect. The billion dollar company who hired Collins in this hypothetical maintainer could be Baidu or Yandex or Saudi Aramco, and then people would be suspicious. It's one thing to have your task be "don't make people mad" but doesn't that change if people getting mad can give you unbounded retrospective work and responsibility?
> "If these tasks fall under the "don't make everyone mad" job responsibility, then yes. [...] Was there actually a real maintenance issue? [...] Or, the community could raise the problem to Big Company, who can do the work of verifying that there is a problem and working with the maintainer to resolve it."
As soon as the internet becomes aware that they can get anything merged ASAP by threatening to get mad, everyone will be mad about everything all the time. Whom at the BigCo will do the work of verifying whether there is a problem? I mean, let's put Lasse Collins on a specific team along with other employees who are expected to work 40-80 hour weeks while he isn't. The pressure on the team manager to ditch the maintainer and distribute his salary among the other team members would be constant. If those other team members see him doing less work for similar or more money it would be a morale killer and they would want to leave. If they also have to know his project well enough to follow all the drama and things people are complaining about and tease out what is and isn't a real problem and coerce him to do his job, sorry 'work with him', well, they won't be very motivated to do that.
The guy in Nebraska is still not getting anything but will also not have the stress of becoming one of the main characters/victims in a huge attack.
In this instance, noticing the people emailing to pressure you have fake looking names that start with adjacent letters and the same domain name.
Be more paranoid.
Does that even matter these days?
Especially if we're talking nation state level stuff convincing histories are not hard to create to deflect casual observers.
>Be more paranoid.
Most people in OSS just want to write some code to do something, not defend the world against evil.
Yes, it would have prevented this attack. It isn’t totally sufficient but it’s quick and easy and would have prevented this attack.
“Most people don’t want …”
I get it. I think the issue is that pushing junk code from malicious contributors into your project causes more hassle in the long run. If you just want to code and make stuff work, you should probably be careful who you pull from. It’s not just for the benefit of others, it’s first and foremost to protect the code base and the time and sanity of other contributors.
The more paranoid walls you put up the more actual contributors getting into the movement say "eh, screw this, who wants to code anyway".
This isn't just a problems with OSS, this is a fundamental issue the internet as a whole is experiencing and no one has good answers that don't have terrible trade offs of their own.
Look at the number of people who make meaningful commits to ffmpeg vs the number of people that wrap it in some UI and somehow persuade thousands of people that their solution is amazing, and ffmpeg is a sexy project.
It's like seeing a "Bill Johnson, real American patriot" account on Twitter: would not believe it at face value.
If my speculation is correct then the the exact date on which access was granted must then first be known, after that a trusted backup of the repo from before that date is needed. Ideally Lasse Collin would have a daily backup of the repo.
Although perhaps the entire repo may have to be completely audited at this point.
I think it's a pretty reasonable assumption that this didn't happen, though it would be nice to see a testimony to that effect from someone trustworthy (e.g. "I restored a xz checkout from a backup taken before 5.6.0 and all the commit IDs match").
Isn't this part just a few pages of code, if that?
I agree that it will be blindingly obvious for the reasons you list.
https://research.swtch.com/xz-timeline#jia_tan_becomes_maint...
would be useful info for Lasse Collins before taking the pressure campaign seriously.
Let's say they have fake passports and physically appear at key signing parties. Now you're screwed because even your peers (that you thought know how to validate identities using passports) will get fooled.
Read more on GPG's trust levels: https://www.gnupg.org/gph/en/manual/x334.html
Can somebody write a script which diffs the release tarballs from the git sources for all debian packages and detects whether there are any differences apart from the files added by autotools :)?
I believe this means it was unauthorized, not unauthenticated.
(https://www.google.com/search?q=%22unauthenticated+remote+co...)
Consider the impact if anybody else, not just this attacker, could have exploited it -- on the other hand, it may have been discovered sooner (had it also not been discovered by accident first).
Remote code execution means a single thing, running JavaScript when accessing a web page and using SSH as intended is not RCE.
The links you point to there are about "RCE attack" which also implies not authorized.
> in the introductory paragraph it reads "unauthenticated, targeted remote code execution ... I believe this means it was unauthorized, not unauthenticated.
You again:
> agree that RCE, unqualified, means unauthorized RCE
So you agree that your "unauthorized" qualification from your orignal post was unwarranted, since all unqualified RCE are unauthorized.
Now if you want to split hairs, you'll say "but the introductory paragraph said unauthenticated, which is qualified RCE, thus I am still right". ok then
All RCEs are classified in either unauthenticated or authenticated, the former being the worst (or best if you're a researcher/hacker).
For something like SSH which has authentication and authorization as features, I would expect to talk about an RCE in that context, and not the backdoor’s auth features.
This backdoor bypasses both authentication (not requiring an account password, authorized key, etc on the target system) as well as authorization (as it doesn’t check a user against any policy for what commands or users can log in).
Ideally, we would have a mechanism to verify that a given build _matches_ the source for a release. Then it wouldn't matter where it was built, we would be able to independently verify nothing funky happened.
One of the best things about Golang is that packages are shared direct via source repositories (Github) rather than a package repository containing mystery tarballs. I understand the appeal of package repositories, but without proper security constraints it's a security disaster waiting to happen.
Github+ even has a scheme for signing artifacts such that you have some level of trust they came from inside their Actions system, derived from some git commit. This would allow the benefits of a modular build for a large product like a distro, while preserving a chain of trust in its component parts.
+Not advocating a dependency on Github per se -- the same sort of artifact attestation scheme could be implemented elsewhere.
From the big ones - 7z, ncurses are both tarballs only.
I would think a "real identity" should be required by linux distros for all /major/ open source projects/library committers which are included in the distro, so that we can hold folks legally accountable
Google's Know, Prevent, Fix blog post floated the idea of stronger identity for open source in https://security.googleblog.com/2021/02/know-prevent-fix-fra... and there was very significant pushback. We learned a lot from that.
The fundamental problem with stronger identity is that spy agencies can create very convincing ones. How are distros going to detect those?
I realize, it's a hard problem. (And, thanks for the link to the "Know, Prevent, Fix" post.)
PS: FWIW, I "win my bread" by working for a company that "does" open source.
Edit: Some projects I know use in-person GPG key signing, or maintainer summits (Linux kernel), etc. None of them are perfect, but raises the bar for motivated anonymous contributors with malicious intent, wanting to become maintainers.
And there are other reasons some people might not want to travel outside their nearby region. For instance, they might be taking care of an elderly relative. Or they might be the elderly relative, with travel counter-indicated for health reasons.
What I'm saying is, in context of critical-path software, the identity of maintainers vs non-maintainers matters more. I'm not naively claiming that it'll "solve" the problem at hand, just that it's another layer in defense. For a critical software, you shouldn't be able to simply submit a "patch"[1] such as:
tests: Add-binary-blob-with-a-subtle-backdoor.xz
Signed-off-by: "Anonymous Rabbit" <LittleBunny123@lolmail.com>
It's a messy situation. How much, if at all, "clever tech" can mitigate this human "trust issue" is an open problem for now.
[1] https://git.tukaani.org/?p=xz.git;a=commitdiff;h=cf44e4b7f5d
But you have a point. As an agency you can seed two jiatan's to serve diligently for a couple of years following the strict 2-person code reviews and then still poison the project. On the other hand, if the xz build process was automated and transparent and release artifacts were reproducible and verifiable even in this poor condition of xz-utils as a project it would have been much harder to squeeze in a rogue m4/build-to-host.m4
Source code is provided without warranty and this statement is clear in the license.
Putting an verified identity behind the source code publish is basically starting to twist said said no-warranty. Fuck that.
It only happened in the last 10 years apparently.
Why do sshd and xz-utils share an address space?
When was the sshd -> systemd dependency introduced?
When was the systemd -> xz-utils dependency introduced?
---
To me this ARCHITECTURE issue is actually bigger than the social engineering, the details of the shell script, and the details of the payload.
I believe that for most of the life of xz-utils, it was a "harmless" command line tool.
In the last 10 years, a dependency was silently introduced on some distros, like Debian and Fedora.
Now maintainer Lasse Collin becomes a target of Jia Tan.
If the dependency didn't exist, then I don't think anyone would be bothering Collin.
---
I asked this same question here, and got some good answers:
https://lobste.rs/s/uihyvs/backdoor_upstream_xz_liblzma_lead...
Probably around 2015?
So it took ~9 years for attackers to find this avenue, develop an exploit, and create accounts for social engieering?
If so, we should be proactively removing and locking down other dependencies, because it will likely be an effective and simple mitigation.
Linking against libsystemd on Debian is about as suspicious as linking against libsystem on MacOS. It's a userspace library that you can hypothetically avoid, but you shouldn't.
As for why systemd links against xz, I don't know, and it's a bit surprising that an init system needs compression utils but not particularly surprising given the kitchen sink architecture of systemd.
It's for the journal. It can be optionally compressed with zlib, lzma, or zstd. That library had not only the sd_notify function which sshd needed, but also several functions to manipulate the journal.
Read the lobste.rs thread for some quotes on process separation and the Unix philosophy.
There are mechanisms other than dynamic linking -- this is precisely the question.
Also, Unix supported remote logins BEFORE dynamic linking existed.
---
What about sshd didn't work prior to 2015?
Was the dependency worth it?
not particularly surprising given the kitchen sink architecture of systemd
That's exactly the point -- does systemd need a kitchen sink architecture?
---
The questions are interesting because they likely lead to simple and effective mitigations.
They are interesting because critical dependencies on poorly maintained projects may cause nation states to attack single maintainers with social engineering.
Solutions like "let's create a Big tech funded consortium for security" already exist (Linux foundation threw some money at bash in 2014 after ShellShock).
That can be part of the solution, but I doubt it's the most effective one.
The problem is the design of rtld and the dynamic linking model, where one shared library can detect and hijack the function calls of another by using the auditing features of rtld. Hardened environments already forbid LD_PRELOAD for injection attacks like this, but miss audit hooks.
My point is that just saying we should use the Unix process model as the defense for supply chain attacks is like using a hammer to fix a problem that needs a scalpel.
> What about sshd didn't work prior to 2015?
systemd notifications, from what it sounds like.
Many mitigations work at the process level, like ASLR, cgroups, and more.
There's a reason that Chrome doesn't allow parsing and rendering in the same process:
https://www.chromium.org/developers/design-documents/multi-p...
Something like the rule of 2 should apply to the sshd -> xzutils dependency -- it's inherently dangerous.
https://chromium.googlesource.com/chromium/src/+/master/docs...
Note that Firefox spent years of engineering effort to catch up with Chrome, retrofitting a multiprocess architecture.
Multiprocess architecture is demonstrably more secure, e.g. if you look at all the actual exploits from https://en.wikipedia.org/wiki/Pwn2Own
Having written multi-process programs used for this purpose, I disagree that it's simple or traditional for the purposes of hardening against supply chain attacks in shared libraries. Only at the most naive level is it appropriate.
And on top of that, it would not have prevented the backdoor. It would only have changed the mechanism used to invoke it.
> If so, we should be proactively removing and locking down other dependencies, because it will likely be an effective and simple mitigation.
I think this has always been important, and it remains so, but incidents like this really drive the point home. For any piece of software that has such a huge attack surface as ssh does, the stakes are even higher, and so the idea of introducing extra dependencies here should be approached with extreme caution indeed.
Or we pay lip service to, but don't use in practice
Notably, Debian bet badly again. They already had this mistake pointed out to them very publicly with the OpenSSL random generator fiasco. Yet they chose to continue applying patches that are not accepted upstream while evidently not understanding the rammifications. Why? Shouldn't there have been a policy change in the Debian project to prevent this from happening again?
I think people are flying past that important piece of the hack. Without that this would not have been possible. If there is a trusted source in the middle building the binaries instead of the single maintainer and the hacker this attack becomes extremely hard to slip by people.
No downstream was using binaries instead of source. Debian and Fedora rebuild everything from source, they don't use the binaries supplied by the maintainer. The backdoor was inserted into the build system.
> One portion of the backdoor is solely in the distributed tarballs
Is it that the tarball created and signed by Jia had the backdoor, but this backdoor wasn't present in the repo on github? And the Debian (or any distro) maintainers use the source code from tarball without comparing against what is in the public github repo? And how does that tarball get to Debian?
The threat actor had signed and uploaded the compromised source tarball to GitHub as a release artifact. They then applied for an NMU (non-maintainer upload) with Debian, which got accepted, and that's how the tarball ended up on Debian's infrastructure.
One part was binary, the test file (pretty common), but checked into the repo. One part was in the build config/script, but was in the source tarball and not in the repo.
I think it's reasonable to think that behind this attack there is a well organized intelligence agency of a big country, so the author can choose the one that they likes more.
Or, even better, delay the issue to the season 2 :D
Generally speaking, you can use the xz story as a background for every type of movie set in the present, an eg.: think about the God's Eye story from the Fast & Furious series. Theoretically, the xz rce can give you more or less the same power God's Eys gives you in the film.
Reflections on Trusting Trust becomes even more interesting if you consider photolithography machines being backdoored.
2024-02-29: On GitHub, @teknoraver sends pull request to stop linking liblzma into libsystemd.[1]
(not in the article) 2024-03-20: The attacker is now a co-contributor for a patchset proposed to the Linux kernel, with the patchset adding the attacker as a maintainer and mirroring the attacker's activity with gaining the trust over the development of xz-utils.
A theory is that the attacker saw the sshd/libsystemd/xz-utils vector as closing soon with libsystemd removing its hard dependency on xz-utils. When building a Linux kernel image, the resulting image is compressed by default with gzip [3], but can also be optionally compressed using xz-utils (amongst other compression utilities). There's a lot of distributions of Linux which have chosen xz-utils as the method used to compress kernel images, particularly embedded Linux distributions.[4] xz-utils is even the recommended mode of compression if a small kernel build image is desired.[5]
If the attacker can execute code during the process of building a new kernel image, they can cause even more catastrophic impacts than targeting sshd. Targeting sshd was always going to be limited due to targets not exposing sshd over accessible networks, or implementing passive optical taps and real time behavioural analysis, or receiving real time alerts from servers indicative of unusual activity or data transfers. Targeting the Linux kernel would have far worse consequences possible, particularly if the attacker intended to target embedded systems (such as military transport vehicles [6]) where the chance of detection is reduced due to lack of eyeballs looking over it.
[1] https://github.com/systemd/systemd/pull/31550
[2] https://lkml.org/lkml/2024/3/20/1004
[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
[4] https://github.com/search?q=CONFIG_KERNEL_XZ%3Dy&type=code
[5] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
[6] https://linuxdevices.org/large-military-truck-runs-embedded-...
If you compile and build your own image, would that be able to trigger the backdoor?
You can of course change an existing image to something that triggers the backdoor, but with that level of access you won't really need a backdoor, do you?
1: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
1. A new "test" is added to the xz-utils repository, and when xz is being built by a distribution such as Debian, the backdoor from the "test" is included into the xz binary.
2. The backdoored xz is distributed widely, including to a target vendor who wishes to from a Debian development environment compile a kernel for an embedded device that is usually used in a specific industry and/or set of countries.
3. When backdoored xz is then asked to compress a file, it checks whether the file is a kernel image and checks whether the kernel image is for the intended target (e.g. includes specific set of drivers).
4. If the backdoored xz has found its target kernel image, search for and modify random number generation code to effectively make it deterministic. Or add a new module which listens on PF_CAN interfaces for a particular trigger and then sends malicious CAN messages over that interface. Or modify dm_crypt to overwrite the first 80% of any key with a hardcoded value. Plenty of nasty ideas are possible.
The way the exploit was setup, they could have pivoted to targeting basically any application server because there's so many interdependencies. python, php and ruby could be targeted because liblzma is loaded via libxml2 as an example.
Gaining trust for linux kernel commits would have just let them continue maximizing profit on their time investment.
>particularly if the attacker intended to target embedded systems (such as military transport vehicles [6])
Said vehicles aren't networked on the public internet and from experience in this particular sector, probably haven't been nor will be updated for decades. "Don't break what isn't broken" applies as well as "military doesn't have a budget to pay defense contractors $1 million to run apt-get update per vehicle".
You are giving a lot of credit to that, I have seen military ones with ancient software, even the “new” updated ones are still on ubuntu 18.04 because of some drivers/sdks compatibility, but it isn’t a major issue since most of the times they are not connected to the public internet.
I'm not a security expert though.
So, like UNIX does have this plumbing, its just that reaching for libraries and tight integration has been the pursuit of Lennart Poopering and his clan for years.
This could be "Denizens":
- an inhabitant or occupant of a particular place.
- a person admitted to residence in a foreign country especially : an alien admitted to rights of citizenship
Good way to describe Jia Tan (singular or plural) and their sockpuppets.
First from the article itself:
>"At this point Lasse seems to have started working even more closely with Jia Tan. Evan Boehs observes that Jigar Kumar and Dennis Ens both had nameNNN@mailhost email addresses that never appeared elsewhere on the internet, nor again in xz-devel."
That is an important observation!
Takeaway: Most non-agenda driven actual people on the Internet, leave a trail -- an actual trail of social media and other posts (and connected friends) that could be independently verified by system/social media/website administrators via voice or video calls (as opposed to CAPTCHA or other computer-based "Is it a human?" tests, which can be gamed) for stronger confidences in the identity / trustworthiness of the remote individual at the other end of a given account...
Next takeaway from the linked article: (https://boehs.org/node/everything-i-know-about-the-xz-backdo...):
>"AndresFreundTec @AndresFreundTec@mastodon.social writes:
Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd,
showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol.
Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates."
Takeaway: We could always use more observability of where exactly CPU time is spent in given programs...
Binaries without symbol tables (which is the majority of programs on computers today) make this task challenging, if not downright impossible, or at least very impractical -- too complex for the average user...
Future OS designers should consider including symbol tables for all binaries they ship -- as this could open up the capability for flame graphs / detailed CPU usage profiling (and the subsequent ability to set system policies/logging around these) -- for mere mortal average users...
Rinse, repeat for all critical-path open source software. A bit like the OpenSSL "Heartbleed" disaster[1]. OpenSSL is now part of Linux Foundation's (they do a lot of great work) "Core Infrastructure Initiative".
Many fat companies build their applications on these crucial low-level libraries, and leave the drudgery to a lone maintainer in Nebraska, chugging away in his basement[2].
Also works like Karen. "this project ...." "ok Jia"
If this compromised OpenSSHd had become the default across millions of systems all over the world, could a worm-like thing have brought a major chunk of the Internet down? Imagine millions of servers suddenly stuck in a boot-loop, refusing to boot.
And all because one owner of a library had some mental health issues. We should not have such SPOFs.
Wrong takeaway.
I can't seem to find it in the (web version of) the maillist.
Another question:
What is the typo exactly in this commit? I can't seem to find it.
https://git.tukaani.org/?p=xz.git;a=commitdiff;h=a100f9111c8...
Hit reply
Attackers find a vulnerable but critical library in the supply chain. Do a cross check on maintainers or owners of the code (reference social media and other sources to get more information). Execute social engineering attack. Talk to maintainer(s) "off list" to gain rapport. Submit a few non-consequential patches that are easy to grep to gain trust. Use history of repository or mailing list against victim to gaslight ("last release was X years ago!1", "you are letting project rOt!", "community is waiting for you!").
To me this is the polar opposite of pig butchering. This was a targeted and unromantic attack, unrelated to investing or cryptocurrency, the original maintainer was not "fattened like a hog" in any way, if anything he was bullied and abused into submission.
Do you think this was a planned effort or was it opportunistic? Did they know what they were doing and social engineered towards it, or did they figure out what to do based on the day-to-day context they discovered?
Just thinking out loud, would it possible to go over with LLM and analyse existing OSS mailing lists and issues to classify such sentiment from the users?
Also check this... https://www.abuseipdb.com/check/64.23.252.16
I have no idea what that IP address is supposed to be about...
Doesn't necessarily validate anything though. Could be progressof planting misdirection given that the IP address only started being detected basically today (and the VPS was likely only just setup today as well, if the hostname is to be trusted).
... and that progressof's account is about an hour old.
> 2024-02-23: Jia Tan merges hidden backdoor binary code well hidden inside some binary test input files. The associated README claims “This directory contains bunch of files to test handling of .xz, .lzma (LZMA_Alone), and .lz (lzip) files in decoder implementations. Many of the files have been created by hand with a hex editor, thus there is no better "source code" than the files themselves.”
This is, perhaps, the real thing we should think about fixing here because the justification is on the surface reasonable and the need is quite reasonable - corrupted test files to test corruption handling.
But there has got to be some a way to express this which doesn't depend on, in essence, "trust me bro" since binary files don't appear in diffs (which is to say: I can think of a number of means of doing it, but there's definitely no conventions in the community I'm aware of).
It sort of makes sense, since at the end of the day it could just be statically linked or implement B's behaviour itself and do whatever it wants, but it's not really what you expect is it.
But in practice that's not something that can be enforced for arbitrary projects without those projects having set something up specifically.
For example, the project could track the effect on binary size of the production binary after every PR. But then it still requires a human (or I guess an AI bot?) to notice that the increase would be unexpected.
The correct way to do that would be a source that generates a test file and then a script which reproducibly produces the desired corruption.
As someone working in security, the fact that _foundational_ pieces of the computing/networking rely on motivated individuals and essentially goodwill is mind blowing.
There are great aspects to the FOSS movement, but the risks – particularly the social engineering aspects as demonstrated here – and potential blast radius of supply chains like this... We take it all for granted and that is lining up to bite us hard, as an industry.
I don't see how this is any strongly different than some unappreciated skill worker in a corporation. Its interesting the double standard we have for FOSS. Meanwhile in the commercial world, supply chain attacks are commonplace and barely solicit headlines.
Yes, FOSS needs to be able to address these kinds of attacks, but the world runs on the efforts of the low-level few, generally. The percent of people who work to build and maintain core infrastructure has always been small in any economic system. The world is held up by the unsung labor of the anonymous working class. Think of all the people working right now to make sure you have clean water, electricity, sanitation, etc. Its a tiny fraction of the people in your city.
Conversely, why aren't all these corporations who depend on this contributing themselves? Or reaching out? There's a real parasitic aspect here that gets swept under the rug too.
I'd even argue this isn't really a hobby for many, especially for higher profile projects. For many its done for social capital reasons to build up one's reputation which has all sorts of benefits, including corporate advancement, creating connections for startups, etc. Its career adjacent. And that's ignoring all the companies that contribute to FOSS explicitly with on-the-clock staff.
So there are motivators more than just "I'm bored and need a hobby." Its a little dismissive to call FOSS development just a hobby. Is what Linus does a hobby? I don't think most people would think so. Things like this have important social and economic motivators. The hypothetical guy in the comic isn't some weirdo doing something irrationally, but has rational motivators.
I'd also argue that its pretty harmful to FOSS adoption if the community takes on a "well, its a hobby don't expect quality, security, or professionalism." This is a great way to chase people away from FOSS. We can't just say "Oh FOSS is better than much closed software" when things are good, then immaturely reply "its just a dumb hobby, you're dumb for trusting me," when things go south. I think its pretty obvious there's a lot of defensiveness right now and people being protective over their social capital and projects, but I think this path is just the wrong way to go.
Comms, PR, and image management in FOSS is usually bad (see Linus's rage, high profile flame wars, dramatic forkings, ideological battles, etc), so optics here aren't great, because optics is something FOSS struggles with. The community is at best, herding cats, with all manner of big personalities and egos, and its usually a bit of a controlled car crash on the best of days.
In a traditional corporation, people would come to an office. It would be known where they live. If you would require something like (code) review, it becomes a lot harder to plant something. Obviously not impossible, but hard for all but the most dedicated attackers.
In contrast, with open source and poorly funded projects. People don't always have money to travel. So the people working on an open source project may only know each other by some online handles. Nerds typically don't like video conferencing. So it is quite possible to keep almost everything about an identity secret.
And that makes it a lot more attractive to just try something. If something goes wrong, the perpetrator is likely in a safe jurisdiction.
Corporate insiders do stuff like this too, its just how often do we hear about it? FOSS has high visibility but closed source doesn't. Think of all the shady backdoors out there. Or what Snowden and others revealed.
On average a 100% FOSS organization is going to be much, much more secure than a 100% commercial close source one. Think of all the effort it takes to moderately secure a Windows/closed source stack environment. Its an entire massive industry! Crowdstrike alone has a $76bn marketcap and that's just one AV vendor!
Commercial software obeys the dictates of modern capitalism. Projects get rushed, code review and security take a backseat to quarterly reports and launch dates, etc. This makes closed source security issues common.
Usually when the exploit is discovered the attacker is far outside the victim's jurisdiction. See all the crypto gangs operating from non-Western non-extradition states.
Pretty much never.
One particular terrible case I saw was when a developer left a testing flag in a build that got pushed to production and used for years. Had you set the right &whatever flag in the URL you'd have unauthenticated access to everything. It was discovered years after the fact when the software was no longer in supported status, so nothing was ever wrote up about it and told to the users. "They shouldn't have been using it by now anyway, no use in bad press and getting users worried".
No time to do things right...
Read up on red teaming and social engineering in general. Many more examples of red teaming are available, for example. I thoroughly enjoy these specific stories on Darknet Diaries podcast.
> We can't just say "Oh FOSS is better than much closed software" when things are good, then immaturely reply "its just a dumb hobby, you're dumb for trusting me," when things go south.
It's weird. There are the explicit expectations of FOSS (mostly just licenses, which say very little), and the implicit expectations (everything else).
It's anarchic and ad hoc in a way that leaves the question of "what are we actually doing with this project(s)" up for all kinds of situational interpretation, as you noted. This is bad, because this ambiguity leads to conflict when the various actors are forced to reveal their expectations, and in doing so show that their expectations are actually quite divergent (i.e., "this is my fun hobby project!" vs. "my company fails without this bugfix!" vs. "I thought this was a community project!" vs. "This project is for me and my company, I call the shots, you're welcome to look at the code, though").
It's a little bit like the companies that are like "we have a flat management hierarchy, no one really reports to anyone else". It's just not true. It's almost always used as a ruse to dupe a certain class of participant that isn't sophisticated enough to know that these kinds of implicit power hierarchies leave them at a disadvantage. There's always a structure, it's just whether that structure is explicit or not. This kind of wishy-washy refusal to codify project roles/importance in FOSS is not doing us any favors. In fact I think it prevents us from actively recognizing the "clean water" role that an enormous number of projects play.
There's real labor power here if we want it, but our continued desire to have FOSS be everything to everyone is choking it.
I'm not knocking FOSS at all. I just think some people have the concept twisted. Just like the meme of being written in Rust means the code is fast/safe from the mere fact it was written in Rust. I don't write Rust, but if I did, I guarantee that just from sheer not knowing WTF I'm doing would result in bad code. The language will not protect me from myself. FOSS will not protect the world from itself, but it does at least allow for decent investigations and after action reports.
Look at stuff getting signed with MS keys, hardware vendors with possible backdoors.
Social engineering is social engineering and it can happen anywhere no matter the profit motivation or lack there of.
Money interest in software won't save you any more than Foss.
The main issue is that this attack shows how a relatively unknown component, as part of a much larger and more critical infrastructure, is susceptible to pressure as a result of "this is a hobby project, lend a hand".
At what point do these components become seen as a utility and in some way adopted into a more mainline, secure, well-funded approach to maintenance? That maintenance can, and probably should, happen in the open, but with the requisite level of scrutiny and oversight worthy of a critical component.
We got very lucky, _this time_.
why just this one? do we collectively have the memory of a gold fish? just recently, log4j had a similar blast radius. is it because one was seemingly malicious that the other doesn't count?
This vulnerability, had all gone to the attackers plan, would have been present on the major distros next major releases through a key infrastructure component which would have been installed far more widely, IMO.
Another major difference is that Log4J is already part of the Apache Foundation, which means it should have greater oversight/security maintenance anyway, while this is an attack against a solo developer.
It's definitely not to downplay the severity of the Log4J incident, by any means. But they are decidedly different.
I would bet that some of these projects like xz would show enormousness benefits from one paid person working on it 1/4 time, leaving room for a couple more projects per dev. Additionally, a couple places providing relatively minor grants would probably help a dev buy back some of their time so the can work on their project some other time then 'after the kids are in bed'
In your Rust example, using C is like using a power tool without any safety measures. That doesn't mean that you are going to get hurt, but there is an expectation that a sizable fraction of users of such tools will get hurt.
Rust is then the same tool with safety measures. Of course it is still a power tool, you can get hurt. But the chances of that happening during normal operation is a lot lower.
I think xz is a good example where open source happened to work as intended. Somebody noticed something weird, alerted and other people could quickly identify what other software might have the same problem.
I don't think that's quite true - maybe a couple horses got out, but this was caught early and did not get to infect very many machines because someone completely unaffiliated could review it and find it.
The scope of "what FOSS needs to be supported and well-funded" just increased by an order of magnitude.
There is no accountability and in fact high-five and star shots for those taking piles of money and placing it on more piles of money, instead of doing what appears to be obvious to almost everyone on this thread -- paying long-term engineers.
This ought to be not only a wake-up call for businesses, but also to hobbyists and members of the public in general; our code, our projects and our social "code-generating systems" must be hardened or face rampant abuse and ultimately be weaponised against us.
In a way, these issues which FOSS is facing and are becoming apparent are no different to those democracy been submitted since time immemorial.
The solution here has to be some combination of a dramatic cut back on the number of individual projects we rely on and well-audited technical solutions that treat all code—even FOSS dependencies—as potentially malicious and sandboxes it accordingly.
The demand for more assistance here is the angle that was played in social engineering, specifically the demand to acquire more maintainers due to workload. Especially if such support would take the form like providing source archives with manipulated build scripts that are rarely checked by third parties.
There is also a problem of badly behaving industry that tries to take control of "hobby projects". Speaking of which, these "hobby"-projects often have much better code than many, many industry codebases.
I think FOSS overall still lessens the risk. It got more risky since it has been integrated in social media since these often allow for developer being shouted down or exploited much more easily.
No is an answer. And blocking people should be a thing.
Personally I do not publish anything anymore which is not "non-commercial only" as a result of demands, you can do it yourself if you want to make money off it (or if you demand things for that matter). Fortunately my online stuff isn't used much but even then it's possible to get "requests".
- Corporate maintainers are great until they enshittify things in the pursuit of profit (see Oracle)
- Nonprofits are probably the best but can go the same route as corps (see Mozilla)
- Hobbyists are great until they burnout (see xz)
It kinda is. The fact that code costs fractions of a penny to copy and scale endlessly, changes everything.
There's hard limits on power plants, you need staff to run them, it's well-understood.
But software - You can make a startup with 3 people and a venture capitalist who's willing to gamble a couple million on the bet that one really good idea will make hundreds of millions.
Software actually is different. It's the only non-scarce resource. Look at the GPL - Software is the only space where communism / anarchy kinda succeeded, because you really can give away a product with _nearly_ no variable costs.
And it's really just the next step on the scale of "We need things that are dangerous" throughout all history. Observe:
- Fire is needed to cook food, but fire can burn down a whole city if it's not controlled
- Gunpowder is needed for weapons, but it can kill instantly if mishandled
- Nuclear reactors are needed for electricity, but there is no way to generate gigawatts of power in a way that those gigawatts can't theoretically cause a disaster if they escape containment
- Lithium-ion batteries are the densest batteries yet, but again they have no moral compass between "The user needs 10 amps, I'm giving 10 amps" and "This random short circuit needs 10 amps, I'm giving 10 amps"
- Software has resulted in outrageous growth and change, but just like nuclear power, it doesn't have its own morality, someone must contain it.
Even more so than lithium and nuke plants, software is a bigger lever that allows us to do more with less. Doing more with less simply means that a smaller sabotage causes more damage. It's the price of civilization.
So the genie ain't going back in. And private industry is always going to be a tragedy of the commons.
I'm not sure what government regulation can do, but there may come a point where we say, even if it means our political rivals freeload off of us, it's better for the USA to bear the cost of auditing and maintaining FOSS than to ask private corporations to bear that cost duplicating each other's work and keeping it secret.
Is that a handout to Big Tech? 100%. Balance it with UBI and a CO2 tax that coincidentally incentivizes data centers to be efficient. We'll deal with it.
You have the right to be paid for your time. It's valuable.
I enjoy coding too... but the only free coding I do is for myself.
Use a proper license, charge for your time and stop killing yourself doing unpaid hobby projects that cause nothing but stress.
Because it never starts that way. It scratches an itch, solves an interesting puzzle and people thank and praise the work. Deep down we all want to be useful, and it helps that it looks great on a résumé.
After it's established the big corps come along, but the feeling of community usefulness remains. It's also why so many devs burn themselves out, they don't want to disapoint.
The work was not being done "for giant corps"; it was being done for everyone, and giant corps just happen to be a part of "everyone", together with small corps, individual people, government, and so on.
> You have the right to be paid for your time. It's valuable.
When you think of free software contributions as "volunteer labor" instead of just a hobby, it makes more sense. Yes, my time is valuable; when I'm working on free software, I'm choosing to use this valuable time to contribute to the whole world, without asking for anything in return.
IF you understand why you'd post without being paid, you're 80% of the way to realizing why people program without being paid.
Here on HN it’s been derided as “company just checking a box for compliance but adds no functionality. It slows us down when we want to disrupt!” - developer of yet another todo list or photo editor app…
Buffer overflows and the like are one thing. Notions from this blog that certain normal files won’t be well reviewed is a bad smell in software. Innocuous files should be just as rigorously reviewed as it’s all part of the state of the machine
“This is how it’s always worked” is terrible justification
Startup script kiddies git pulling the internet, and single maintainers open source projects aren’t cutting it; if it’s that important to the whole those in charge of the whole need to make sure it’s properly vetted.
I’m an EE first; this really just makes me want to see more software flashed into hardware.
Deloitte coming in a checklist would have NEVER stopped this one.
If that's too much bureaucracy or people fear that governments might exert undue influence: hand the money to universities, go back to the roots - many (F)OSS projects started out in universities after all. Only issue there is that projects may end up like OpenStack in the end ;)
Are you trying to pull an xz attack on me?
The xz attack involves, in significant part, a maintainer burned out and happy to accept offered help. I don't think making it substantially harder to receive genuine help is likely to improve the situation.
Also, you are forgetting bringing up 'are you trying to pull xz to me?' is a yellow flag towards the person who said it. It isn't definite, it just makes people raise up, it gathers the attention of watchful eyes. We should be careful not overdoing it though.
This is of course based on the fact of how the xz attack needed a couple of apparently innocent community members to start being too pushy, to the point of almost bullying the author and pointing fingers to their supposed passivity towards maintaining the project.
To some levels of such behavior, after all that has happened, one could reasonably (even if just jokingly) wonder if it's not a similar attempt. IMO, being framed as a possible attacker would probably either calm the shit out of some too entitled users, or provoke them into even more whining.
Now seriously, there is the thing about attitude that I wish was more popular among FOSS maintainers in order to have a mentally healthy relationship with their role: the ability to work on their own terms and not forget even for a split second that this is all a hobby activity. That FOSS licenses are explicitly written to allow A LOT of freedoms such as hiring 1st or 3rd party support services, forking, or whatever else but not to demand anything from the author.
Do we know who really is Jia Tan? Any photo of him? The email addresses he has been using? His location?
2022-06-08: Lasse Collin said that he had worked off-list a bit with Jia Tan.
I think that "off-list" means not on the xz mailing list, maybe over the personal email/phone/voice app. That's why I thought that Jia Tan was a real person
1. It seems a lot of people are assuming Jia Tian was compromised all along. I haven't seen a reason to believe this was a long play, but rather that they were compromised along the way (and selected for their access). Why play the long game of let's find a lone package, w/ a tired maintainer, and then try to get in good with them vs. Let's survey all packages w/ a tired maintainer, and compromise a new entrant (ie bribe/beat them).
2. IMO this also is a failing on behalf of many government agencies. NSA, NIST, FBI all should, IMO spend less time compromising their own citizenry, and more time focusing on shoring up the security risks Americans face.
Here's the thing, if the maintainer was so tired and needed the help, the pressure is more risk than reward. The maintainer would be relieved to have help... But pressure risks estranging...
To be clear I'm aware this is just my own musing though.
They don't seem to exist outside of this incident and things related to this.
There were multiple people who also don't seem to exist outside of their posts to the xz mailing list applying pressure for the original maintainer to bring Jia on board. This occurred around the time that Jia was first making contact with the project, not only recently.
Apparently the IP addresses that were logged for Jia is a VPN based in Singapore
They have vanished.
Honestly, there's very little evidence that they weren't always intending this.
https://boehs.org/node/everything-i-know-about-the-xz-backdo...
IRC activity
I just found this: https://news.ycombinator.com/item?id=39868773 which is definitely not where I originally read it, but libera is probably ultimately the source
So I assume Jia has spoken out since? How do you go this long without realising someone else is making plays as you?
I also think it is more difficult to get away with bribing/beating an existing contributing than you suggest; esp since failure means likely exposure.
Everyone wants to consume it. Nobody wants to participate.
People are upset when a company like Elastic or Mongo switches to a "non open" license. But at the same time, the market doesn't leave much choice. Companies won't be incentivized to contribute to projects when they can freeload. The market actually wants vendors, it doesn't want to participate in open source. But they don't want to _pay_ for vendors.
So I think its entirely appropriate that anyone / any entity that creates "open source" to change their license, set limits, say "no", and let users be damed unless they're willing to make it financially appealing. It's literally "Without Warranty" for a reason.
Letting your passion project becoming hijacked into determining your mental health is really depressing. F' the people who can't get on board with your boundaries, etc around it. They deserve the natural consequences of their lack of support.