The xz attack shell script
research.swtch.com
research.swtch.com
You can write pretty straightforward readable bash, just as I'm sure you can write pretty gnarly python. Especially if you're intentionally trying to obfuscate.
But all this relies on one specific thing about the autotools ecosystem - that shipping the generated code is considered normal.
I know of no other build system that does this? It feels weird, like shipping cmake-generated makefiles instead of just generating them yourself, or something like scons or meson being packaged with the tarball instead of requiring an eternal installation.
That's a lot of extra code to review, before you even get to any kind of language differences.
That's why you don't commit auto-generated code. You commit the generating code, and review that.
Same reason we don't stick compiled binaries in our repositories. Binary executables are just auto-generated machine code.
Well, there's your problem. If you have unreviewed code, anything can be snuck in. Doesn't really matter too much where in your system the unreviewed code is.
> It's normal to have uncommitted autogenerated unreadable shell code in the tarball.
You need to review everything that goes into the tarball. Either directly, or indirectly by reviewing the sources it gets built from. (And then making sure that your build process is deterministic, and repeated by a few independent actors to confirm they get the same results bit for bit.)
Something built on python won't cover quite as wide a range of (obsolete?) hardware.
Of course, you could do the detection logic with some autotools-like shenanigans, but then crunch the data (ie run the logic) on a different computer that can run reasonable software.
The detection should all be very small self-contained short pieces of script, that might be gnarly, but only produce something like a boolean or other small amount of data each and don't interact (and that would be enforced by some means, like containers or whatever).
The logic to tie everything together can be more complicated and can have interactions, but should be written in a sane language in a sane style.
On a slightly different note, unless the application is written in python, it grosses me out to think of writing scripts in python. IMHO, if the script is more complex that what bash is good at (my general rule of thumb is do you need a data structure like an array or hash? then don't use bash), then use the same language that the application is written in. It really grosses me out to think of a rails application with scripts written in python. Same with most languages/platforms.
Writing anything in C is a bad idea these days, and requires active justification that only applies in some situations. Essentially, almost no new projects should be done in C.
Re-doing your build system, or writing a build system for a new project, counts as something new, so should probably not be done in C.
In general, I don't think your build (or build system) should necessarily be specified in the same language as most of the rest of your system.
However I can see that if most of your system is written in language X, then you are pretty much guaranteed to have people who are good at X amongst your developers, so there's some natural incentive to use X for the tooling, too.
In any case, I would mostly just advice against coding anything complicated in shell scripts, and to stay away from Make and autotools, too.
There are lots of modern build systems like Shake, Ninja, Bazel, etc that you can pick from. They are all have their pros and cons, just like the different distributed version control systems have their pros and cons; but they are better than autotools and bash and Make, just like almost any distributed version control is better than CVS and SVN etc.
I use bash habitually, and every time I have an inscrutable or non-intuitive command, I pair it with a comment explaining what it does. No exceptions.
I also don't clean up after scripts for debuggability. I will offer an invocation to do the cleanup though after you've ascertained everything worked. Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing.
There was a lot of intentionally obfuscatory measures at play here and tons of weaponization of most conscientious developer's adherence to the principle of least astonishment, violations of homoglyphy (using easy to mistake filenames and mixed conventions), degenerative tool invocations (using sed as cat), excessive use/nesting of tools (awk script for the RC4 decryptor), the tr, and, to crown it all, malicious use of test data!!!
As a tester, nothing makes me angrier!
A pox upon them, and may their treachery be returned upon them 7-fold!
If your hammer is repurposed shoe, it's fair to blame the tools.
A good practice, but not really a defense against malice, because if the expression is inscrutable enough to really need a comment, then it's also inscrutable enough that many people won't notice that the comment is a lie.
This fundamental limitation on info prop will never disappear. There is nothing harder to do than to legit get somebody to actually read code.
1. Languages where obfuscation techniques look a lot like business as usual.
2. Languages where obfuscation techniques look weird compared to business as usual.
The presence or lack of comments in #1 situations won't really help to bridge the gap.
And the other major issue here, is that xz is a basic system unit, often part of a bare bones, ultra basic, no fluff, embedded linux deployment where other higher level languages likely wouldn't be. It makes sense for the tools constituting the build infra to be low dependency.
And yes. In a low dependency state, you have to be familiar with your working units, because by definition, you have fewer of them.
Unironically, if more people weren't cripplingly dependent on luxuries like modern package managers have gotten them accustomed to, this all would have stuck out like a sore thumb, which it still did once people actually looked at the damn thing*.
Also, Python build scripts can be living hell too, full of dancing devils that could be introducing backdoors left and right - just look at your average Conan recipe, particularly for larger/more sensitive libraries, like OpenSSL or libcurl.
And you can't really name such small functions well anyway, not when they're broken down for the sake of being small. Case in point, some build script I saw this week had function like `rename_foo_dll_unit_tests` calling `rename_foo_dll_in_folder` calling `rename_foo_dll` calling `rename_dlls`, a distinct call chain of four non-reused functions that should've been at most two functions.
Are all Python build scripts like that? Not really. It's just a style I've seen repeatedly. The same is the case with inscrutable Bash scripts. I think it speaks more about common practices than the language itself (notwithstanding Bash not really being meant for writing longer programs).
The way it works is that you can provide "recipes", which are Python scripts, that automate the process of collecting source code (usually from a remote Git repository, or a remote source tarball), patching it, making its dependencies and transitive dependencies available, building for specific platform and architecture (via any number of build systems), then packaging up and serving binaries. There's a lot of complexity involved.
Here are the two recipes I mentioned:
libcurl: https://github.com/conan-io/conan-center-index/blob/master/r...
OpenSSL v3: https://github.com/conan-io/conan-center-index/blob/master/r...
Now, for the sake of this thread I want to highlight three things here:
- Conan recipes are usually made by people unaffiliated with the libraries they're packaging;
- The recipes are fully Turing-complete, do a lot of work, have their own bugs - therefore they should really be treated as software comonents themselves, for the purpose of OSS clearing/supply chain verification, except as far as I know, nobody does it;
- The recipes can, and do, patch source code and build scripts. There's supporting infrastruture for this built into Conan, and of course one can also do it by brute-force search and replace. See e.g. ZLib recipe that does it both at the same time:
https://github.com/conan-io/conan-center-index/blob/7b0ac710... -- `_patch_sources` does both direct search-and-replace in source files, and applies the patches from https://github.com/conan-io/conan-center-index/tree/master/r....
Good luck keeping track of what exact code goes into your program, when using Turing-complete "recipe" programs fetched from the Internet, which fetch your libraries from somewhere else on the Internet.
I don’t think these issues would necessarily be solved at all by waving a hand and replacing it with similarly complex build tools. Bazel, for example, can be a daunting tool to fully grasp. Any tool used should be well understood. Easier said than done of course.
And shell has a lot of footguns.
That said, what can help drastically here are well-defined best practices and conventions built into the language which, admittedly, bash really doesn’t have.
- Mutating default arguments to functions, so subsequent calls have different behavior
- Somewhat particular rules around creating references vs copies
- Things that look like lambda captures but aren’t quite
- Bash: installed everywhere you want to work (yes, who actually wants to work on Windows ;-)
- C/C++: when speed/size matters there was no alternative except Assembly until recently
- Javascript: until recently this was the most sane option for client side code on the web (Active X and Java applets existed yes but managed to be even worse.)
- PHP: Low cost hosting, Function-As-A-Service way before that became popular, shared nothing architecture, instant reload for local development bliss
- cannot know which object attributes are private or public (and some classes use settable properties so you can't say "just don't set any attributes on non-dataclass objects")
As long as you apply the same standards to what seems to be everyones darling: Javascript.
Javascript has the same amount of footguns as PHP and Bash but has gotten away with it by being cute (and having a whole menagerie if support tools around it to make it possible for ordinary people to write workable code in it).
(Yes, I am qualified to rant about Javascript BTW. I wrote a working map rendering system with pan and zoom and automatic panning based on GPS location using ECMAScript and SVG back in the spring of 2005. I think roughly half a year before Google Maps became public. Back before all the modern JS tooling existed. All I had was JEdit with syntax highlighting. Perl at least let me put breakpoints in my code even back then.
And yes, I have written more JS since then.)
Now you can at least just stick to React and TypeScript and bundle it using Webpack and have months of relative sanity between each time you have to throw something out and replace it.
-> REPL essentially for free (the language IS a REPL)
-> enormous installed base
-> No compilation (well, unless you have something--like autotools--using shell as essentially a transpilation target)
-> No need for "libraries" in most cases: the ordinary CLI that $vendor already ships can be used right away, with no need for a custom SDK for whatever "real" language you would otherwise be using. For example, if you are already familiar with the "aws" CLI program, it's trivial to treat it as an "API" for a quick shellscript instead of needing to dig into the boto3 docs to do something equivalent the "right" way.
-> Pretty good integration with standard *nix facilities (redirecting STD{ERR,OUT}, checking existence of files/pipes, checking/setting exit codes, etc.)
I might agree with "it's not hard to learn it", but I don't agree with "it's not hard to remember it".
I really don't understand this point, its a script language, how old is it doesn't make any difference. I've come accross some Powershell scripts that were unreadable down to its verbosity with certain things, and if you don't already know all the flags and options for it, it's hopeless to try and understand.
Both serve a purpose, neither are 'evil'.
When you use long options in bash (scripts), it becomes very readable, but it's not a widespread practice. I always use long options while writing scripts.
Consider these two examples, which is very straightforward:
- curl -fsSL $URL | bash
- curl --fail --silent --show-error --location $URL | bash
The second one almost "talks you through".
It's true that Bash and Perl has one of the most contractible syntax around, but it's not impossible to make it more understandable.
However, these parts of codebases are considered "supportive" and treated as second class citizens, and never receives the same love core parts of the codebases enjoy. That's a big mistake IMO.
When you make something more readable all around, hiding things becomes harder exponentially.
I challenge you to find a single easily readable >100 line bash script and link it here (I do think small scripts can be fine).
I may add more if I have time today.
Who says I blindly curl2sh these scripts? I download them, read them and understand what they do before running them actually.
Long options make obscure command lines readable and understandable, actually, and if you vet them, you understand what they are doing.
Short options make things harder because I need to open the man of the command to understand what it is doing.
You don't have to do everything blindly as the quick start documents outline.
WTFing at shell scripts is normal.
"WTFing". This is brilliant.
This isn’t just because I’m more familiar with Python. I don’t even think it’s the main reason. It’s just that Python is more likely to be able to be read in a ‘natural language’ sort of way. It’s better at doing what it says on the tin. It’s more able to be read via pure intuition by a programmer that’s not familiar with Python specifically.
In bash land? “What the hell is [[?”
And yes, I could come up with 20 ways off the top of my head that Python is a far-from-perfect language.
And I’m not even saying that Python is the right tool for the job here. Maybe we’re better off with one of the many more modern attempts at a shell language. The main thing is that we should acknowledge that bash has overstayed its welcome in many of the areas in which it’s still used.
The reality is that we as a society have made meaningful progress as far as designing readable languages goes. And we LITERALLY have many orders of magnitude more resources to make that happen. That’s something to feel good about. It’s unreadable to continue to mischaracterise some sysadmin greybeard’s’ familiarity with bash as an indication that it is in any way intuitive or readable.
Like, sheesh, now we all sound like C developers trying to justify an absurdly footgun-laden standard library just because we happen to know the right secret incantations, or think that we know them, anyway. But now this is definitely becoming a religious war…
(Though not sure how fool-proof these are.)
And not all languages are Turing complete in the first place. Not even all useful languages.
Bash is great at dealing with files, text, running other programs, job handling, parallelism and IPC.
Those things in combination can end up being more complex in Python, which creates more opportunities for obfuscation.
And type safe.
autoconf creates a shell script by preprocessing with m4. So you need to know not just the intricacies of shell scripting, but also of m4, with its arcane rules for escaping: https://mbreen.com/m4.html#quotes
If autoconf used m4 to generate python scripts, they would also look like https://pyobfusc.com/#winners
That's AI phase one
Hey, I phase 2 is even better disguised exploit code hiding behind acres of seemingly plausible AI generated code
I haven’t used it for some time but autoconf always seemed like a horrible hack that was impossible to debug if it didn’t work properly.
That was bad enough back in the days where one was mostly concerned with accidents, but in more modern times things that are impossible to debug are such tempting targets for mischief.
I don't know why.
Same reason why I don't like TypeScript in its current form. It's not worth the extra complexity it brings.
And how did we get to the point that a complex system is required to build a compression library -- something that doesn't really have to do much more than math and memory allocation?
The project in question contained a compression library, but was not limited to it; it also contained a set of command line tools (the "xz" command and several others).
And a modern compression library needs more than just "math and memory allocation"; it also needs threads (to make use of all the available cores), which is historically not portable. You need to detect whether threads are available, and which threading library should be used (pthreads is not always the available option). And not only that, a modern compression library often needs hand-optimized assembly code, with several variants depending on the exact CPU type, the correct one possibly being known only at runtime (and it was exactly in the code to select the correct variant for the current CPU that this backdoor was hidden).
And that's before considering that this is a library. Building a dynamic library is something which has a lot of variation between operating systems. You have Windows with its DLLs, MacOS with its frameworks, modern Linux with its ELF stuff, and historically it was even worse (like old a.out-based Linux with its manually pre-allocated base address for every dynamic library in the whole system).
So yeah, if you restrict yourself to modern Linux and perhaps a couple of the BSDs, and require the correct CPU type to be selected at compilation time, you could get away with just a couple of pages of simple Makefile declarations. But once you start porting to a more diverse set of systems, you'll see it get more and more complicated. Add cross-compilation to the mix (a non-trivial amount of autotools complexity is there to make cross-compilation work well) and it gets even more complicated.
The last build system I worked on separated build and test as separate stages. That meant you got a lot of useless artifacts pushed to a development namespace on the distribution server, but it also meant later stages only needed read access to that server.
>No one has any business saying they know what something does until they've actually read it.
Beneath the placid surface of abstraction is the den of the devil.
Also the basic workflows for the alternative build systems have maybe ten more characters to type. It's not bad.
The designer of Meson explicitly avoided making the language turing complete so for example you cannot define functions. In my experience this was an excellent decision to limit people tendency to write complex stuff and put the pressure on the Meson developer to implement themselves all the useful functionalities.
In my experience the Meson configuration are as simple as they can be and accommodate only a modicum of complexity to describe OS specific options or advanced compiler option one may need.
Please note that some projects' Meson file have been made complex because of the goal to match whatever the configure script was doing. I had in mind the crazy habits of autotools to check if the system has any possibly used function because some system may not have it.
Also, meson's build dependencies (muon, python) are a lot for some of these projects.
I worry about all the con/pro-sumer hardware makers, from Philips Hue to Alexas, from the SumUps to the camera makers, from Netgear to TP-Link. All their products are packed with open-source libraries. And I am 100% certain that most of their dev teams do not spend time scanning these for obscure injection vectors.
This rationale baffles me, it feels that the dependency-hell circlejerk crowd is working on making OSS maintainers look even more bad with this scenario.
Any given commercial operation that claims any credibility for itself does supply chain analysis before adopting a dependency. This is, among other things why ordinarily you'd pay RedHat to maintain a stable Linux Release for you and why projects such as FreeBSD severely limit the software they ship in the default install.
If you are affected by this mess, I'm sorry to say, but it's your fault. If you are worried about developers of software you use for free, as in free beer, going rogue, either put in incentives for them to not do that (i.e. pay them) or fork the project and implement your own security measures on top of what's already there.
If you're worried that you could encounter exploits from dependencies in commercial software you use, you should negotiate a contract that includes compensation from damages from supply chain attacks.
If you're unwilling to do that, sorry mate, you're just unprofessional.
Inb4: Yes, I am really trying to say that you should check the supply chain of even your most basic dependencies such as SSH.
Just look at stuff like pip, npm or pretty much any "modern" package manager in use by developers - they're all pretty much designed to pull in a shitload of arbitrary unaudited and in some causes unauditable dependencies.
And nobody wants to listen. That's why I prefer to work in heavily regulated areas nowadays - that way I can shorten that discussion with "yeah, but regulatory requirements don't let us do that, sorry"
The absolute basic should be having a local archive of dependencies which at least received a basic sanity check, and updates or additions to that should review changes being added. CI gets access to that cache, but by itself does not have network access to make sure no random crap gets pulled into the build. You'd be surprised how many popular build systems can't do that at all, or only with a lot of workarounds.
You can also setup your own dependency registry and only work with that.
No they're not. The dependency circle jerk went so far to prompt NPM to display all subsequent dependencies on each libraries page.
The issue lies with the industry as a whole exploiting the work of OSS developers for their own gain and having the audacity to complain when these volunteers won't additionally do a security audit for free.
There are already lots of groups maintaining internal lists and analysis of dependencies they trust. If there was a platform for reporting safety, rather than reporting vulnerability, one could say "Only allow packages that someone from a fortune 500 company publish an analysis of".
That sounds like you assume RedHat would've caught the vulnerability in xz-utils, before shipping it in the next release of RHEL. I'm not so sure about that, as there is only so much you can do in terms of supply chain analysis and such a sophisticated vulnerability can be pretty hard to spot. Also mind that it only got discovered by accident after all.
It really sounds like you’re speaking from the perspective of a hypothetical employee looking to not get PIPd or whatever.
GP is talking about something quite different, and you’ve run off taking some sort of great personal offence to someone dare implying that there are downsides to open source, not even that it’s worse overall, but that there are downsides.
Chill.
I feel this is the crux of it for the thread. Most places where I've worked have been regulated and this has been interested to read/follow.
This 'fall on the sword' thing is real. The 'engineer on a PIP' thing is too, in a twisted sense. This has multitudes/depth.
Consider business terms/liability. Your certification/ability to do business depends on implementing certain things, sometimes by buying things (ie: RHEL) from those who also carry certifications. The alternative is to do it yourself at great expense.
If 'it' hits the fan, you can [hopefully] point at due diligence. It's not an engineer doing this to cover themselves... but businesses.
I don't know how approachable the distribution providers are as a smaller business. We, at fairly large enterprises, were able to work closely with them to get fixes regularly - but that says very little.
Anyway: I say all this to neither defend or deride the situation. It's sort of like a cartel, insurance, and buying merch for a band on tour, all in one.
I've benefited from this situation but also lost years of my life to it
I'm not saying its not a companies problem if this exploit got into their RHEL environments. But from a company perspective when it comes down to law suits, they will get to shift the blame to RHEL. And for a business, that is what matters. Do you really think companies care about having secure systems? I would be willing to bet money, if companies could be protected from lawsuits from data breeches, they wouldn't give two shits about security. For them, data breeches are just potential multi-million or multi-billion dollar legal liabilities. And this is part of RHEL's business model. You get to shift some of that legal liability to RHEL.
The organisations that did have some measures were complained about loudly, and they diluted their requirements over time in order to avoid stagnation. Example: Debian used to have a "key must be signed by three other Debian developers" requirement. They had to relax the requirement in part because, from the perspective of the wider ecosystem, nobody else had these onerous requirements and so they seemed unreasonable (although Covid was the final straw). If we'd had an ecosystem-wide culture of "know your upstream maintainer", then this kind of expectation as a condition of maintainership would be normal, we'd have much better tooling to do it, and such requirements would not have seemed onerous to anyone. It's like there's an Overton Window of what is acceptable, that has perhaps shifted too far in favour of velocity and at the cost of security, and this kind of incident is needed to get people to sit up and take notice.
This incident provides the ecosystem as a whole the opportunity to consider slowing down in order to improve supply chain security. There's no silver bullet, but there are a variety of measures available to mitigate, such as trying to know the real world identity of maintainers, more cautious code review, banning practices such as binary blobs in source trees, better tooling to roll back, etc. All of these require slowing down velocity in some way. Change can only realistically happen by shifting the Overton Window across the ecosystem as a whole, with everyone accepting the hit to velocity. I think that an individual or organisation within the ecosystem isn't really in a position to stray too far from this Overton Window without becoming ineffective, because of the way that ecosystem elements all depend on each other.
> If you're unwilling to do that, sorry mate, you're just unprofessional.
There are no professionals doing what you suggest today, because if they did, they'd be out-competed on price immediately. It's too expensive and customers do not care.
Applies to all of my devices btw. I don't like Android having to use an old kernel, I didn't like MacOS running some ancient Darwin/BSD thing, etc. The required effort for backporting worries me.
Don't get me wrong, I'm not saying OSS has no vulns.
One opposite of this I've seen is Mikrotik's RouterOS. I'm under the understanding that they usually reimplement software and protocols rather than depending on an upstream.
I'd imagine that is what leads to issues such as missing UDP support in OpenVPN for 10 years, and I'm not sure it gives me the warmest fuzzy feeling about security. Pros and cons, I suppose. More secure because it's not the same target as everybody else. Less secure because there are fewer users and eyes looking at this thing.
Using the absolute latest version is acting as a beta tester for everyone else and this is not the first case where it means you get absolutely hosed.
We should probably be more afraid of the backdoors you can’t see in proprietary that would almost never be found.
On the contrary: the developers and maintainers who are more informed than us described it as highly sophisticated attack. I also read early InfoSec (information security) articles which were able to only describe a part of the code, not the whole strategy behind the attack because, again, the attack and code are sophisticated. You can also read early InfoSec articles which describe the attack in different ways simply because it was not that simple to understand. Then I read articles saying something like this: "Finally it seems it's an RCE attack".
Of course, now that even a scanner is developed to detect that vulnerability on your server, we can all claim: "Oh that was a so simple and stupid attack, how come no one detected it much earlier ?!"
For example oss-fuzz was building xz by cloning the github repo directly. There was never a chance for oss-fuzz to discover the backdoor, because only the tarball had it, not the repo itself. So that oss-fuzz PR might genuinely just be a genuine thing unrelated to the backdoor.
>2024-02-29: On GitHub, @teknoraver https://github.com/systemd/systemd/pull/31550 to stop linking liblzma into libsystemd. It appears that this would have defeated the attack. https://doublepulsar.com/inside-the-failed-attempt-to-backdo... that knowing this was on the way may have accelerated the attacker’s schedule. It is unclear whether any earlier discussions exist that would have tipped them off.
I'm the author of such PR. My purpose was to trim down the size of the initram files by removing unneeded dependencies.
I couldn't imagine that liblzma had a backdoor.
But many others followed, like libselinux: https://github.com/systemd/systemd/pull/19997
libqrencode: https://github.com/systemd/systemd/pull/16145
p11kit: https://github.com/systemd/systemd/pull/25771
tpm2-util: https://github.com/systemd/systemd/pull/28333
libiptc: https://github.com/systemd/systemd/pull/29836
libkmod: https://github.com/systemd/systemd/pull/31131
Exactly during the development of the libkmod PR, someone noted that libxz could be lazily loaded too: https://github.com/systemd/systemd/pull/31131#issuecomment-1...
And so I proposed myself to to the job, nothing less, nothing more.
If you look at the code of the other PRs, you see that they are very very similar, there are also macros to easy this task, like DLSYM_FUNCTION()
To be more precise, the first public comment asking to dlopenify lzma was dated 30 Jan by Daan: https://github.com/systemd/systemd/pull/31131#issuecomment-1...
The day after, it was reiterated by Lennart: https://github.com/systemd/systemd/pull/31131#issuecomment-1...
But if you look in the systemd repo there is a TODO file with a section of libraries which needs to be lazy loaded. liblzma was added in this list in June 2020 (https://github.com/systemd/systemd/commit/cdfd853744ee934869...) by Lennart, and removed by me just after that my PR was merged.
> Many of the files have been created by hand with a hex editor, thus there is no better "source code" than the files themselves.” This is a fact of life for parsing libraries like liblzma. The attacker looked like they were just adding a few new test files.
Yes, these files are scary, but I can see the reason. But at least can we keep them away from the build?
> Usually, the configure script and its support libraries are only added to the tarball distributions, not the source repository. The xz distribution works this way too.
Obligatory auto tools wtf aside, why on earth should the tarballs contain the test files at all? I mean, a malicious test could infect a developer machine, but if the tars are for building final artifacts for everyone else, then shouldn’t the policy be to only include what’s necessary? Especially if the test files are unauditable blobs.
Last time we were doing that we were preferring git upstream, though, and generated autocrap as needed - I never liked the idea of release tarballs containing stuff not in git.
The repeated check is indeed mysterious. My only hypothesis is that the attacker may have thought that it should look plausible as a test input to a compression library, hence repetition.
The malicious code that the xz backdoor inserts into the library is a compiled x86_64 object file so it only is targeting one platform.
Or just add some laziness.
One thing I noticed while trying to reproduce the exact bytes included in the .xz file committed to git is that the script's xz stream doesn't seem to be compressed by any of the default xz presets, I was only able to reproduce it by using `xz --lzma2=dict=65536 -c stream_2`. All the default numbered presents chose a different dictionary size. Another odd seemingly intentional choice, but again I don't understand the reasoning.
Still not sure about the repeated lines, though now I'm convinced there must be some reason for it.
It's pretty amazing how the tools keep up with the increasing complexity of the products we make.
And to be honest, in most cases just make it simpler, I think people just don't like to learn new stuff.
This is why open source will always outperform closed source in terms of security. Sure it pointed a massive flaw in the supply chain. Sure it highlights how under appreciated the foundational elements of FOSS, leaving maintainers subject to manipulation.
But the same attack within a private company? Shit, probably wouldn't even need advanced obfuscation. With a large enough PR and looming deadlines, could easily sneak something like this with a minimal amount of effort into production systems. By the time company even realizes what happens, you are already flying off to a non-extradition country and selling the exfiltrated data on Tor (or dark web).
Even countries we have strained relationships with will extradite as part of a negotiation when it's convenient for them politically.
Russia probably wouldn't have even kept Snowden if it wasn't state secrets he revealed. If it was just some random data breach they would have prisoner-swapped him for an oligarch caught money laundering elsewhere.
As someone old enough to remember the tail end of the early hacker eras (e.g. Mitnick), I don't think anyone SHOULD be extradited over this, in particular if they're not being charged with actually using the exploit. Prosecute them where they live. Should they be prosecuted 193 times over in every state on Earth? What's the nexus? Github? Every server that installed the compromised xz utils?
But you are right they will deport (not extradite) foreigners who are inconvenient to them or when it is politically expedient to do so, if the foreigners are a nuisance, or as part of a political negotiation or prisoner exchange.
The whole "extradition treaties" meme is a misconception. You will only get extradited if you flee to a country where you are a citizen (even dual citizen), or the ability to assert citizenship/nationality there. A fugitive fleeing to a country without an extradition treaty is subject to deportation. Every country on earth reserves the right to deny entry to or deport foreign fugitives. They might choose not to if someone is found to be a refugee, subject to the death penalty in a non-death-penalty state, etc.
Several countries do not extradite their own citizens. For citizens of these countries, going back to their own home country would be enough.
When you get hired to a private company, they know who you are. That’s an immediate deterrent against trying anything funny. On Github, no one knows who you are. It might be harder to backdoor a project without getting noticed, but there is no risk to getting noticed. You can try as many times as you like. Jia Tan is still at large, and didn’t even have to plan their whole life around living in a non-extraditing country (if they aren’t in one already).
Happens all the time. Maybe it's a state actor. Maybe it's a disgruntled employee. It's just not in the same lens as you expect (software supply chain attack).
Apple has trouble keeping the lid on top secret projects. Leaks about designs happen all the time prior to scheduled debut at WWDC.
MS has had trouble in the past as well when it came to developing the Xbox (One?).
"Owners of China-Based Company Charged With Conspiracy to Send Trade Secrets Belonging to Leading U.S.-Based Electric Vehicle Company" - https://www.justice.gov/usao-edny/pr/owners-china-based-comp...
"Ex-Google engineer charged with stealing AI trade secrets while working with Chinese companies" - https://www.latimes.com/world-nation/story/2024-03-07/ex-goo...
"The US Hits Huawei With New Charges of Trade Secret Theft" - https://www.wired.com/story/us-hits-huawei-new-charges-trade...
"U.S. charges China-controlled company in trade secrets theft" - https://www.pbs.org/newshour/economy/u-s-charges-china-contr...
"Ex-Google and Uber engineer Anthony Levandowski charged with trade secret theft" - https://www.theverge.com/2019/8/27/20835368/google-uber-engi...
In the case of Levandowski, the dude didn't even bother with covering his tracks. Straight up just downloads trade secrets from source control and transfers them to personal computer - https://www.justice.gov/usao-ndca/press-release/file/1197991...
In this small sample of cases, were the exfiltration attempts as elaborate as the "xz attack"? Probably not, but all of these people were vetted by internal procedures and that did nothing to stop them from acting maliciously.
Forget back dooring the project when getting through the front door is so much easier! People are very relaxed in their walled off garden and cubicle :)
Has Jia Tan been "charged" with anything?
And then, once you're in, you cannot just commit to your target project willy-nilly, as your manager and your manager's manager will have other priorities. A for-profit company's frequently dysfunctional management would actually work as deterrent here: you don't just need to justify your code, you will have to justify why you were working on it in the first place.
A smooth talker can get you to relax your guard.
Identity can be faked, especially if you have a nation state backing you.
Also, that's still one named person that no one would like to end up being, so that alone acts as a deterrent.
I suspect a "professional job seeker" with the resources of the NSA behind them and who lands 1st and subsequent interviews dozens of times a year - would be _way_ better at landing interviews and jumping through stupid recruiting hoops that even the best senior or "10x engineers", who probably only interview a dozen or two times in their entire career.
This would be impossible to enforce, and might not be a good idea because it enables other ranges of attacks: if you know the identities of the maintainers of critical open source projects, it’s easier to put pressure on them.
If someone makes a library and other people start using it, are they forced to reveal their identity?
Do the maintainers get paid?
That's not going to help, and will be fairly easy to circumvent for nation state actors or similar advanced persistent threats who will not have a problem adding an extra step of identity theft to their attack chain, or simply use an agent who can be protected if the backdoor is ever discovered.
On the other hand, the technical hoops required for something like that will likely cause a lot of damage to the whole open source community.
The solution here is learn from this attack and change practices to make a similar one more difficult to pull off:
1. Never allow files in release tar-balls which are not present in the repo.
2. As a consequence, all generated code should be checked in. Build scripts should re-generate all derived code and fail if the checked in code deviates from the generated.
3. No inscrutable data should be accessible by the release build process. This means that tests relying on binary data should be built completely separately from the release binaries.
Part of the appeal of cyber operations for intelligence agencies is that there's basically no tradecraft involved. You park some hacker in front of a laptop within your territory (which also happens to have a constitution forbidding the extradition of citizens) and the hacker strikes at targets through obfuscated digital vectors of attack. They never go in public, they never get a photo taken of them, they never get trailed by counterintelligence.
If you start telling people who want to be FLOSS repo maintainers that they'll need to be at a few in-person meetings over a span of two or three years if they want the keys to the project, that hacker has a much harder job, because in-person social engineering is hard. It has to be the same person showing up, time after time, and that person has to be able to talk the language of someone intimately familiar with the technology while being someone they're not.
It's not a cure-all but for supply chain attacks, it makes the operation a lot riskier, resource-intense, and time-consuming.
You are excluding a ton of contributors based on geography and income.
It's not common that I find this line actually decent but check your privilege with this kind of comment.
This is really a small step away from segregation.
1. Many important contributors, especially in security, prefer to be pseudonymous for good reasons. Insisting on identity drives them away.
2. If a spy agency was behind this, as many people have speculated, those can all manufacture "real" identities anyway.
So you'd be excluding helpful people and not excluding the attackers.
Driver’s license, SSN, national ID, passport, etc. If the government is in on it then there’s no limits.
The only way would be to require physical presence in a trusted location. (Hopefully in a jurisdiction that doesn’t belong to the attacker…)
And in the event someone is pedantic enough to actually care: that person will be considered a pariah that all development stifles due to.
Tensions with the team for nitpicking etc;
FD: I have a situation like this now, I am not the one being picky- one of the developers I hired is. I had to move him out of the team because unfortunately his nitpicky behaviour was not well regarded. (he also comes from eastern europe and has a very matter-of-fact way of giving feedback too which does not aid things).
Both can be fun, but as you probably already know there is a balance.
There are occasional shining lights of stuff done right but I think it's fair to say they're comprehensively outcompeted.
I don't have much experience in low-level optimization, but would a modern CPU not be able to predict the path taken by a brach that tests the CPU features.
> but would a modern CPU not be able to predict the path taken by a brach that tests the CPU features.
That's true, but the CPU has finite branch predictor state, and now you've wasted some of it. Indirect calls hurt too, especially in you need retpolines.
This is a great read: https://www.agner.org/optimize/microarchitecture.pdf
The Linux kernel has interfaces for doing the same thing, more explicitly than ifunc:
If I had a project to develop a backdoor to keep persistent access to whatever machine I wanted, it would make sense that I would have a plug-in executable that I would use for multiple backdoors. That's just decent engineering.
Matrix: #xz-backdoor-reversing:nil.im
IRC: #xz-backdoor-reversing on irc.oftc.net
Discord: https://discord.gg/XqTshWbR5F
- "auto" tools are installer tools and, after all, do good job. Maybe we can download any project that use it, run it once, parse results and put it in /etc ? - becouse it's already there. Then next step: build just-detection tools. Side note: just straight 'make all' will detect it too :) And it show additional next problem that auto's already solved - setting up all that inc/lib paths...
- m4 should be replaced and not by monstrosity like cmake
- GNU's came after MIT's and BSD's, evolved from them. But corporations adapted and assimilated. We need next evolution step: something that forces corporations to spend money on what they use, fund devs or participate in development. OSlicensecertification.orgs should help with that instead of helping corps to be predators becouse some outdated bullshit talks
- yea, auto* tools are way overcomplicated
- finally do that TTY 2.0 !
The most common vulnerabilities are due to how input is handled (or not). XSS vulns, SQL injections, other injections, etc.
Now, the AI agent itself also produces untrusted input. But it does so, in a way, as an extension of the user using it who is actually a trusted entity. Because of this, it is solely up to the user to validate everything coming out of the AI but most users will not do that thoroughly.
What you mention in your comment is a good example of that. If the AI reproduces malicious code e.g. inspired from this campaign, it is essentially an injection attack where the user misses it and didn't properly validate the untrusted input.
Obfuscated autoconf changes, multi-stage deployment, binary blobs (hello lib/firmware and friends, various other encoders/decoders, boot logos, etc), repo ownership changes, new-ish prolific committers in understaffed dependency libraries, magic values introduced without explanation.
What do you mean by APT? If you mean Debian's package manager, that's not what this attack was. This was done upstream and affected non-apt distros just as much.
It's true that upstream is part of apt's supply chain but focussing on apt is misleading.
edit: why the downvotes? I get from the responses that I was wrong but given how the exploit was initially found in a Debian system and a lot of people very quickly jumped on the “Debian patched a thing and broke security” bandwagon, I don’t think it was much of a leap to wonder if that’s what was meant.
Acronyms and initialisms are not the best way to convey specific information.
Advanced, persistent threat.
What are some simple tweaks the "Debians" of the world can do to mitigate this kind of stuff?
Not trust "hand-curated-by-possibly-malicious-maintainers" GitHub release tarballs, only trust git commits?
Whitelist functions that are allowed to do IFUNC/ELF hooking to core OpenSSH functions?
Now realize, those are people having FUN. What is your chance of catching nation state level maliciousness in a codebase? Pretty low.
folders containing test files could easily be removed before build just by name or should live in a separate repository altogether.
If you want to get paranoid about a test subdirectory, stash the payload in comments or formatting choices in the source code itself.
It's great to say "aha, this particular exploit would be deleted by not having binary files in a test directory!", but that totally misses the point that you can put the bytes you want in other places, such as in the source code which is itself a load of bytes. See also polyglot programs and the code is data premise.
Committing binary files to source control rather than including build commands to generate the files is a bit of a red flag.
A good reviewer should have, even for a binary test case, asked the submitter to simplify it to the smallest or most basic binary required for the functionality.
Additionally, a complex example may require multiple conditions to fail and if those aren't split into multiple tests then subtle bugs can be reintroduced later because that complex test doesn't cover all potential failure conditions. If there need to be test cases for multiple related bugs then they need to be minimal to demonstrate the failure condition combinations they are testing for.
How could it play any role here? It doesn't bring any dependencies, does it?
As an aside, C/C++'s header system with conditional inclusion is also fundamentally broken. Even templates are just text substitution with a thin veneer of typing.
I think about Google's build system, which is very much designed to avoid this kind of thing. The internal build tool is Blaze (Bazel is the open-source cousin). Many years ago, you could essentially write scripts in your BUILD files (called genrules) that were hugely problematic. There was no way to guarantee the output so they had to be constantly rebuilt. There was a long project to eliminate this kind of edge case.
Blaze (and Bazel) are built around declaring hermetic units to be built with explicit dependencies only. Nothing shipped to production is built locally. It's all built by the build servers (a system called Forge). These outputs are packaged into Midas packages ("MPMs"). You could absolutely reconstruct the source used to build a particular library, binary or package as well as the build toolchain and version used. And any build is completely deterministic and verifiable.
C/C++, Make, CMake, autotools, autoconf and all that tooling so common in Linux and its core dependencies absolutely needs to go.
Nix at the very least provides first-class(-ish) build support for modern languages like Rust, Go and Python, but I don't think anyone has written an actual Nix builder for C/C++. A combo of Bazel + Nix is fairly common, though.
IMO, it's hard to say if Nix would "solve this attack", since we've only seen it being truly used on more modern things where the build complexity of the actual piece of code is not much more than the same few commands.
As for pulling down a poisoned tarball, I think the discussion here is rather about upstream projects using Nix to guarantee reproducible builds rather than Nix trying to fend off attacks downstream in Nixpkgs. In modern Nix (with Flakes), this would look something like being able to clone and run `nix build .#` and end up with the same output every time.
Ideally nix would move towards less imperative/genrule style package declarations and ultimately become more usable for internal builds.
Wouldn't it be more accurate to say something like "older build systems"? I don't think any of the things you listed are "modern". Which isn't a criticism of their legacy! They have been very useful for a long time, and that's to be applauded. But they have huge problems, which is a big part of why newer systems have been created.
FWIW, I have been using pants[0] (v2) for a little under a year. We chose it after also evaluating it and bazel (but not nix, for better or worse). I think it's really really great! Also painful in some ways (as is inevitably the case with any software). And of course it's nearly impossible to entirely stomp out "genrules" use cases. But it's much easier to get much closer to true hermeticity, and I'm a big fan of that.
I was under the impression that Blaze is Google's Bazel deployment, i.e. that they're the same code. Is that not correct?
There are a few gaps I think xz-utils highlights:
- Repositories containing a mixture of source code, build scripts, test frameworks, static resources and documentation generation scripts are all considered to be a single security domain with no isolation between them. If you look to Gentoo's src_prepare function as an example, we perhaps should instead split this into build_src_prepare, doc_src_prepare, test_src_prepare and install_src_prepare instead. If source code is being built and binaries generated, the sandboxed build directory should perhaps not contain test files and documentation generation scripts. If the package is being installed with "make install" (or equivalent) then static resources (such as a timezone database) should be available to copy to /usr/, but build scripts used to generate the binaries or documentation do not need to be available to "make install" (or equivalent).
- Sandboxing used for package building hasn't traditionally been implemented for security reasons in the past. Sandboxing should perhaps be hardened further with modern and more complex approaches such as seccomp to further protect build systems from the likes of libbackdoored that are targeted towards package maintainers/Linux distribution build systems. As a further example to seccomp, Gentoo's 'sandbox' has Linux namespace isolation built in, but not yet enabled whilst it is tested.
- There is a lack of automated package management tools (including dashboards / automatic bug creation) for comparing source trees in Git to released tarballs and making discrepancies more visible and easier for maintainers to review.
- There is a lack of automatic package management tools (including dashboards / automatic bug creation) for detecting binary and high entropy files in source trees and confirming they are validly formatted (e.g. invalid tag in a TLV file format) and confirming that test and example files contain nothing-up-my-sleeve content.
There has already been an accelerated move in recent years towards modern and safer build systems (such as meson and cargo) as 80's/90's C libraries get ripped out and replaced with modern Rust libraries, or other safer options. This is a lot of work that will take many years though, and many old 80's/90's C libraries and build systems will be needed for many more years to come. And for this reason, sandboxing/isolation/safety of old build systems seemingly needs to be improved as a priority, noting that old build systems will take years or decades to replace.
[1] https://devmanual.gentoo.org/general-concepts/sandbox/index....
[2] https://devmanual.gentoo.org/ebuild-writing/functions/src_pr...
This was all possible because XZ, despite being effectively everywhere, has one actual part-time maintainer who had other things going on in his life. That also means that there's nowhere near enough resources to redo the build system to some more secure alternative. If they had, say, 10 enthusiastic and skilled volunteer contributors with plenty of free time, they could do that, but then a new person appearing out of nowhere with a few helpful commits would never have a chance at being made an official maintainer or sneaking sketchy tools and code past the other maintainers.
Not that I'm blaming XZ or the real maintainer. Clearly whoever was behind this was looking for the weakest link, and if it wasn't XZ, it would have been something else. The real problem is the culture.
So I guess what this really means is someone at a big corporation making Linux distros should audit their full dependency tree. Any tool in that tree that isn't actively maintained, say at least 3 long-time active contributors, they should take over one way or another - whether that's hiring the current maintainer as a full-time remote employee, offering to buy out the rights, or forking and running under their own team.
I'm not necessarily super thrilled with that, but I guess it's the world we live in now.
However I've seen way too many projects where individuals in the 'team' are able to carve out impenetrable fiefdoms where they can operate with wide latitude.
I could see a Jia Tan being able to pull this off in a team context as well - bigger teams might even be weaker. (Everyone welcome Jia - he's going to write test cases and optimize our build process so everyone can focus on $LAUNCH_DAY)
This is it precisely. When you're paying Redhat for an "enterprise" Linux then that guarantee should extend down their entire software stack. Just getting the odd backported patch and so-so email support no longer cuts it.
- bug fixes (as every non-trivial software has bugs)
- improved security (as the kernel adds security functionality (capability dropping, sandboxing, ...), software can utilize this functionality to reduce its attack surface)
- improvements of existing features (e.g. utilizing new CPU extensions or new algorithms for improved performance)
And if you confuse C with C++ and Make with CMake/Autotools/Autoconf, you have a lot to learn. Look: simple and portable makefiles work:
git://bitreich.org/english_knight