The xz sshd backdoor rabbithole goes quite a bit deeper
twitter.com
twitter.com
I'm not saying it's amateur-ish to have bugs. I'm saying, if this was developed by a highly competent state-sponsored organization, you'd think they would have developed the actual exploit and tested it heavily behind closed doors, fixing all of the bugs and ensuring there were no suspicion-creating performance regressions before any of it was submitted into a public project. If there was no performance regression, much higher chance this never would have been discovered at all.
I'm not sure if it's been revealed yet what this thing actually does, but it seems like all it really needs to do is to check for some kind of special token in the auth request or check against another keypair.
> With the backdoored liblzma installed, logins via ssh become a lot slower.
> Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked. This appears to be part of some countermeasures to make analysis harder.
Performance was much worse but not enough to actually nudge the author into digging into it until the random ssh logins started piling up:
https://twitter.com/AndresFreundTec/status/17741907437768663...
https://nitter.poast.org/AndresFreundTec/status/177419074377...
[1] https://www.openwall.com/lists/oss-security/2024/03/29/4
It happened to be noticed by someone doing benchmarking, but there is another category of machines out there it would hit hard: low end VM's running an ssh server. I've had my low end VM's hit by exactly the same thing the postgres benchmarker saw, but on these box's it isn't just annoying. There are so many ssh logins happening the OOM killer comes out to play but these ssh logins are tiny so it tends to kill the real stuff you've got the box doing, like web serving.
A 2 or 3 fold increase in CPU would suddenly bring a lot of VM to their knees. At least thousands of them. It would be noticed.
The backdoor relies on sshd being patched to depend on libsystemd to call sd_notify(), which several distros had done.
OpenSSH has since merged a new patch upstream that implements similar logic to sd_notify() in sshd itself to allow distros to drop that patch.
So the attack surface of both sshd and libsystemd has since shrunk a bit.
I remember when we added sd_notify support to our services at work, I was wondering why one would pull in libsystemd as a dependency for this. I mean, there's a pure-Python library [1] that basically boils down to:
import os, socket
def notify(state=b"READY=1"):
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
addr = os.getenv('NOTIFY_SOCKET')
if addr[0] == '@':
addr = '\0' + addr[1:]
sock.connect(addr)
sock.sendall(state)
[1]: https://raw.githubusercontent.com/bb4242/sdnotify/master/sdn...
There's also a pure C library (libsystemd itself) which already does all that, and you don't need to test all the error handling cases in your 50 lines of C code. It makes sense to use the battle-tested code, instead of writing your own.
The better question though is...okay, what if the code involved was not simple? xz is a full compression algorithm, compressors have been exploit vectors for a while, so rolling your own is a terrifically bad idea in almost all cases. There's plenty of other more sophisticated libraries as well where you could've tried to pull the exact same trick - there's nothing about it being a "simple" inclusion in this case which implies vendoring or rolling your own is a good mitigation.
The saying goes that everyone is always preparing to fight the last war, not the next (particularly relevant because adversaries are likely scouring OSS looking for other projects that might be amenable to this sort of attack - how many applications have network access these days? An RCE doesn't need to be in sshd).
Writing proper error handling in C is a very tedious and error prone task. So it doesn't surprise me that people would rather call another library instead.
Which shall be harder to justify now: "You're calling a gigantic library full of potential security holes just to call one function, to save writing a few lines of code, are you trying to JIA TAN the project?".
Managing C dependencies is even more tedious and error prone. And even in C, opening a UNIX domain socket to write a single packet is not that hard.
Compare with how s6 does the same thing: instead of passing a random path via an environment variable, it opens the file descriptor itself before exec()ing the binary. All the binary has to do is write a newline to that fd and close it.
No 50 lines of C code needed.
damn it, autocorrect!
Maybe another backdoor, or alternative access mechanism they were using, got closed and they wanted another one in a hurry.
[1] https://en.wikipedia.org/wiki/Poisoning_of_Alexander_Litvine...
Or maybe the opportunity window for the mechanism this backdoor would use was closing. According to the timeline at https://research.swtch.com/xz-timeline there was a github comment from poettering at the end of January which implied that the relevant library would be changed soon to lazy load liblzma ("[...] Specifically the compression libs, i.e. libbz2, libz4 and suchlike at least. And also libpam. So expect more like this sooner or later."), as in fact happened a month later. The attacker had to get the backdoor into the targeted Linux distributions before the systemd change got into them.
Of course, the attacker could instead take the loss and abandon the approach, but since they had written that amount of complex code, it probably felt hard to throw it all away.
At that point it would have been clear “the race is on” to avoid detection, so it’s not too surprising someone would work late to salvage the operation.
Out of interest I looked up the other commit at that time of day visible in that graph, laying on the arrow. It's [1], which changes the git URL from git.tukaani.org to github.com. Of course, moving the project hosting to github was part of the attack.
[1] https://git.tukaani.org/?p=xz.git;a=commitdiff;h=e1b1a9d6370...
Is not possible the attacker simply took over the account of some one genuinely getting involved in the community either hacked or just with $5 wrench and then committed the malicious code ?
Given the behavior of the accounts that applied pressure on the original xz maintainer, this seems unlikely to me.
Sure, in hindsight that's a "duh, why didn't we think of that" - but also it's not very hard at all to see why they didn't think of that. They were likely testing against the system images they were hoping to compromise, not joe-schmoe developer's custom image.
This thing wasn't around for very long but yet another thing to consider would be to keep it working across multiple versions of OpenSSH, libcrypto etc.
> if you aren't able to build TextSecure from source, you probably aren't capable of managing the risks associated with 3rd party sources.
Which is a compelling argument from my perspective. I also think that people who can’t compile code should probably not root their phone.
Further, I don't think it applies to the F-Droid maintainers, who routinely build hundreds of different Android apps for all our benefit. They even directly addressed his concerns about the signing key(s) and other issues by improving F-Droid and met with continued rejection.
Let's never forget that the google play store requires giving google the ability to modify your app code in any way they want before making it available for download. Oh sure, that backdoor will never be abused.
Cute how it's labeled "Danger Zone". So official Signal provided install methods include Google Play Store, or enabling third party APKs and downloading directly from Signal. How the second differs from an official Signal provided and signed F-Droid repository in Moxie's mind is anyone's guess.
What Signal _does not_ allow are APKs built by third parties being distributed under the Signal name, or connecting to Signal servers. Which calls into question the build process itself - the very thing exploited in the XZ backdoor. One either trusts Signal to build the software without backdoors, or doesn't use Signal at all. There is no allowed in between.
Nothing about the way Signal currently does things prevents this from happening today.
Disallowing third party builds only serves to reduce eyes on the build tooling, which we've learned is a great place to hide backdoors.
Equating F-Droid with hackers.ru is a distasteful strawman. F-Droid appear to run as transparent and credible a distribution as Debian or Fedora. Credible enough that the Tor project distributes it's privacy-focused software via F-Droid.
https://nordvpn.com/blog/fbi-honeypot/
Signal could do more to be open with the build process, but opening the door to third party clients is opening the door for APTs to release backdoored Signal clients.
> but opening the door to third party clients is opening the door for APTs to release backdoored Signal clients.
Signal's source code is already public. APTs (or anyone who doesn't care about violating laws) can already produce and disseminate their own builds. There are no technical protections in place to stop them - nor do I know of any which could. The only people who can't currently distribute their own builds are the law abiding good guys trying to build secure software distributions. I'm not sure why you're confused about this, but your assertion that Signal making legal allowances for third party builds adds anything to the capabilities of APTs demonstrates a misunderstanding of what is already available and the (strictly legal) limitations Signal has placed on 3rd parties with regard to distributing independently verifiable builds.
Please take some time to read and understand the github issues, instead of continuing to assert falsehoods or introduce strawmen.
Getting Signal from anywhere else other than them opens up the door for someone to sneak in some code. I am not, in any way, insinuating that fdroid would intentionally do such a thing.
Incorrect. See David A. Wheeler's seminal paper https://dwheeler.com/trusting-trust/
An easy way to avoid talking about strawmen is to avoid bringing one into the conversation. Something to think about.
It's kind of similar to stuxnet but attacking Linux distros is so broad and has such a huge risk of being exposed, as it was within a few weeks of deployment. A good nation state attack would put more effort into not being caught.
But we don't know. So maybe I'm wrong.
Whereas given the number of identities and time involved, the thing we really see is "it took what, 2-3 or burner email accounts and a few dozen hours over 2 years to almost hack the world?"
The entire exploit was within the scope of capability of one guy. Telling ourselves "nation-state" is pretending there isn't a really serious problem.
so this has to be a coordinaed teamwork instead of a single hacker, right?
NSO Group built a turing-complete VM out of a use-after-free exploit in some JBIG2 decompression code. Uploading a payload to the world wide web and calling it bad-3-corrupt_lzma2.xz is clownshoes by comparison.
My best guess as to what this is is an amateur hacker ring, possibly funded by a ransomware group.
[0] https://securelist.com/operation-triangulation-the-last-hard...
[1] https://news.ycombinator.com/item?id=38783112 - HN discussion
The one thing which most leads me to believe this was an intentional backdoor? The S-boxes.
If you look at that HN discussion, you'll find a link to a Mastodon post from an Asahi Linux developer explaining that these "S-boxes" are actually an ECC calculation, and that the registers are probably cache debug registers, which allow writing directly to the cache bypassing the normal hardware ECC calculation, so you have to do the ECC calculation yourself if you don't want a hardware exception caused by an ECC mismatch on read (of course, when testing the cache, sometimes you do want to cause an ECC mismatch, to test the exception handling).
Thankfully! At least in a democracy, the government is chosen, megacorps are accountable to no one else.
https://en.m.wikipedia.org/wiki/Apple–FBI_encryption_dispute
The quality of work you attract in part depends on how much you pay. Go check out how much is paid for a persistent iOS exploit, compared to a Linux user space exploit. From that, you may draw conclusions about their relative perceived difficulty and desirability. This will explain why iOS exploits are done more professionally. They are rarer, much better paid, and thus attract a better audience and more work on guarding them from discovery.
While the world is trying to understand the backdoor, you sir decided that it's "clowshoes". I can only blindly defer to your expertise... "Clownshoes, amateur, hacker ring, ransomeware group." Done.
> Uploading a payload to the world wide web and calling it bad-3-corrupt_lzma2.xz is clownshoes by comparison.
It has to be on the world wide web for distros to package and ship it. And this was actually the best disguise possible: this directory is one where it is normal and expected to have binary files that are not obviously analyzable, as this one wasn't -- another part of the malware rearranged it to become non-corrupt at exploitation-time.
See this note from the README for the test directory:
> This directory contains bunch of files to test handling of .xz, .lzma (LZMA_Alone), and .lz (lzip) files in decoder implementations. Many of the files have been created by hand with a hex editor, thus there is no better "source code" than the files themselves.
It is a brilliant solution to the problem of "okay, but where do I hide the malware payload, given the constraint that it has to be distributed alongside the code and tarballs?". The attack was detected, but not because of this file, and it's unlikely to me that it ever would have been detected purely by the means of this file, given the comment above.
What we have is an Illuminati that is using Jira. :(
You can't conjure quality from nothing (especially if it's pure patriotism/jingoism), large organizations are bound to work with mediocrities and dysfunctional processes, geniuses don't scale. (I feel like stating the obvious)
not sure if we should easily judge offsec and the private-public partnership that provides intel and offensive capabilities.
whether they're ex-criminals or must be accused of "dubious morals" would depend whether their clients (or targets) are what one considers the enemy.
and what about the guy who silently works on a "dubious project" patiently for years ... and then, at the right moment, knowingly throws a spanner in the works? aren't they the true hero?
But wouldn't they be detected at some point by someone? Or would they be silently removed again after some time so the attackers are not revealed?
You have to remember there is a lot of code in our systems. And almost every developer is inadequately trained on how to write secure software.
I wonder what our success rate is in identifying industrial spies? 50%?
Any moderately competent developer could gain maintainership of a huge percentage of open source projects if they’re being paid to focus solely on that goal… after all, they’re competing mostly against devs working on it part side or as a hobby.
If this is state sponsored, they likely have similar programs in a large number of other projects.
Many hackers who work for nation states also have side gigs as crimeware/ransomgang members or actual pentesting jobs.
Reminds me of apt3/boyusec:
https://www.bleepingcomputer.com/news/security/chinese-gover...
It still boggles my mind that americans are against banning companies like huawei and bytedance. The MSS and PLA don't mess around.
The problem is that many others would have as much reason to bann US companies. I mean the US has a much more extensive history of using their security apparatus both for intelligence and economic means even against their allies.
Now if everyone bans everyone else we will let the world economy grind to a halt pretty quickly.
Which is a very different question in practice
"Yes, immigrants are protected by the U.S. Constitution.
The brief answer is “Yes.” When it comes to key constitutional provisions like due process and equal treatment under the law, the U.S. Constitution applies to all persons – which includes both documented and undocumented immigrants – and not just U.S. citizens. Outside the context of immigration policy, the Constitution limits government power over individuals but this does not mean that constitutional rights are absolute."
https://pennstatelaw.psu.edu/sites/default/files/Are%20Immig...
I don't get the point if the mental gymnastics here. We both know of an active threat to americans by a foreign entity. Companies are not people and being able to create and operate a company is not a protected right. Matter of fact, regulating interstate commerce is an explicit right of the government.
For example, you can't transport alcohol across state lines. The government doesn't need a good reason for that regulation, it's their constitutional right.
In this case, simply reciprocating china's bans on US companies would have done the trick.
Honestly, the US government should move to ban all trade with China within a decade or so.
As for bytedance, the government is not claiming chinese immigrants can't own it, they are claiming that the China based owners of bytedance have to sell their stake in the company since any company in China is under the influence of the MSS as evidenced by many examples, the boyusec/apt3 example I mentioned being one.
We don't know shit.
SSO are made by humans under real constrainsts in time, money, personal. It is almost impossible to make something perfect that nobody in the world could detect.
In the world there are people with different backgrounds that could use techniques that you never accounted for. Maybe is a technique used in biology to study DNA, or for counting electrons in a microscope or the background radiation in astronomy.
For example, I have seen strange people like that reverse engineer encrypted chips that were "impossible" to in a very easy way because nobody expected it. They spend 10million dollars protecting something and someone removed the protection using $10.
All of the testing and development that goes into avoiding bugs has a tendency to run counter to the goals of secrecy or speed.
Secret, fast, well tested. Choose 2.
Two years are enough to make yourself familiar with open ssh, ifuncs etc.
Then you do silly things like "hey um I need to replace the bad test data with newly generated data, this time using a fixed seed so that they are reproducible", but you don't actually tell anyone the seed for the new data. Then you lol when that gets past the maintainers no questions asked.
In the end they maybe just wanted to troll a coworker, like play some fart noises while they listen to music, and since they use Debian well, you better find a way to backdoor something in Debian to get into their machine.
Like back in the day when sasser sabotaged half the internet and "security experts" said they have a plausible lead to Russia – which as is turned out was because said security experts ran strings on the binary and found Russian text – put there by the German teen who wrote sasser "for teh lulz".
Patrick McKenzie observes elsewhere that criminal orgs operate more or less like non-criminal ones — by committee and by consensus. By that light, it’s not so hard to fathom why this ship could have been sunk by performance degradation arising from feature bloat. Companies I’ve worked at have made more amateurish mistakes.
Kevin Beaumont speculated [1] that "Jia Tan" saw this and immediately realized it would have neutered the backdoor and thus began to rush to avoid losing the very significant amount of work that went into this exploit; I think he's right. That rush caused the crashing seen in 5.6.0 and the lack of polish which could have eliminated or reduced the performance regressions which were the entire reason this was caught in the first place; they simply didn't have the time because the window had started to close and they didn't want all their work to be for nothing.
[0]: https://github.com/systemd/systemd/pull/31550
[1]: https://doublepulsar.com/inside-the-failed-attempt-to-backdo...
[1] https://ogusers.gg/ is the largest clear net marketplace for buying and selling usernames on popular sites
* The highly professional outfit simply did not see teknoraver's commit to remove liblzma as standard dependency of systemd build scripts coming.
* The race was on between their compromised code and that commit. They had to win it, with as large a window as possible.
* This caused the amateuristic aspects: Haste.
* The performance regression is __not__ big. It's lucky Andres caught it at all. It's also not necessarily all that simple to remove it. It's not simply a bug in a loop or some such. If I was the xz team I'd have enough faith in all the work that was done to give it high odds that they'd get there before discovery. That they'd have time; months, even.
* The payload of the 'hack' contains fairly easy ways for the xz hackers to update the payload. They actually used it to remove a real issue where their hackery causes issues with valgrind that might lead to discovering it, and they also used it to release 5.6.1 which rewrites significant chunks; I've as yet not read, nor know of any analysis, as to why they changed so much. Point is, it's reasonable to think they had months, and therefore, months to find and fix issues that risk discovery.
* I really can't spell this out enough: WE GOT REALLY LUCKY / ANDRES GOT LUCKY. 9 times out of 10 this wouldn't have been found in time.
Extra info for those who don't know:
https://github.com/systemd/systemd/commit/3fc72d54132151c131...
That's a commit that changes how liblzma is a dependency of systemd. Not because the author of this commit knew anything was wrong with it. But, pretty much entirely by accident (although removing deps was part of the point of that commit), almost entirely eliminates the value of all those 2 years of hard work.
And that was with the finish line in sight for the xz hackers: On 24 feb 2024, the xz hackers release liblzma 5.6.0 which is the first fully operational compromised version. __12 days later systemd merges a commit that means it won't work__.
So now the race is on. Can they get 5.6.0 integrated into stable releases of major OSes _before_ teknoraver's commit that removes liblzma's status as direct dep of systemd?
I find it plausible that they knew about teknoraver's commit _just before_ Feb 24th 2024 (when liblzma v5.6.0 was released, the first backdoored release), and rushed to release ASAP, before doing the testing you describe. Buoyed by their efforts to add ways to update the payload which they indeed used - March 8th (after teknoraver's commit was accepted) it was used to fix the valgrind issue.
So, no, I don't find this weird, and I don't think the amateurish aspects should be taken as some sort of indication that parts of the outfit were amateuristic. As long as it's plausible that the amateuristic aspects were simply due to time pressure, it sounds like a really bad idea to make assumptions in this regard.
True in the contents of sshd logins it isn't that big, but ~500ms to get from _start() to main() isn't small either, compared to the normal cost of that phase of library startup. Their problem was that the sshd daemon fork+exec's itself to handle a connection, so they had to redo a lot of the work for each connection.
I suspect they started off with much smaller overhead and then it increased gradually, with every feature they added, just like it happens with many software projects. Here's the number of symbols being looked that a reversing effort has documented: https://github.com/smx-smx/xzre/blame/ff3ba18a39bad272ff628b... https://github.com/smx-smx/xzre/blob/ff3ba18a39bad272ff628bb...
Afaict all of this happens before there's any indication of the attacker's keys being presented - that's not visible to the fork+exec'd sshd until a lot later.
They needed to some of the work before main(), to redirect RSA_public_decrypt(). That'd have been some measurable overhead, but not close to 500ms. The rest of the startup could have been deferred until after RSA_public_decrypt() was presented with something looking like their key as part of the ssh certificate.
The valgrind fix in 5.6.1 overwrites the same test files used in 5.6.0 instead of using the injection code's extension hooks. This is done with what should have been a highly suspicious commit: https://github.com/tukaani-project/xz/commit/6e636819e8f0703... - this replaces "random" test files with other "random" test files. The state reson is questionable to begin with but not including the seed used when the the purpoted reason was to be able to re-create the files in the future is highly suspicous. This should have raised red flags bug no one was watching. I'd say this is another part of the operation that was much more sloppy than it needed to be.
> almost entirely eliminates the value of all those 2 years of hard work.
Except control over xz-utils/liblzma would have still been very valuable even without the sshd exploit path as it's central use in the toolchain used to build Linux distributions would have allowed for many other attacks.
Libsystemd was moving to a dlopen architecture for its dependencies.
This means that the backdoor would not load as the sshd patch only used libsystemd for notify, which does not need liblzma at all.
So they IMHO gave it a last shot. It's OK if it burns as it would be useless in 3 months (or even less).
The collateral is the backdoor Binary, but given enough engineering power it will be irrelevant in 2-3 months, too.
The only thing that makes me think this was amateurs/criminals instead of professionals is that I tend to think that professionals are more interested in post attack security.
So if the gate was closing an amateur would say, "Act now! Let's get what we can!" A professional would say, "This is all going to come to light real soon - our exploit won't be read and there's a high chance of this all falling apart. Pull out everything, cover our tracks to the degree we can and find another opportunity to pull this off."
But then again I also think on professionals would work an exploit that takes years. Criminals by their nature want a quick payout (If I had the patience for a long con I'd just get a job) a motivated individual amateurs (i.e. crazy people) rarely have a wide enough skill set.
So the systems developer job is to build a reputation over years as we see, the security guy's job is to handle the equation group heavily encrypted/obfuscated exploits.
This would explain a mix of code quality / professionalism - multiple people, multiple roles. Of course the former "systems programmer" role need not be a full time employee. They good have motivated a white hat developer to change hats, either financially or through an offer they can't refuse.
Always possible that was "parallel construction" evidence.
Someone at a TLA discovered the attack by some other means, had a quiet Signal chat with a former colleague who works at MS...
> Classic conspiracy theory.
I would not be too quick to shit on "conspiracy theories" however as there are plenty of proven cases of people conspiring against the interests of the public.
Though I guess you end up with some other questions if it's totally anonymous. But I often will do a quick look over commits of things that I upgrade (more for backwards compat questions than anything but)
Is it that they just got unlucky to get caught, or is this type of attack just too hard to pull off in practice?
I’d like to think the later. But, we really don’t know.
This may be confirmed by regular vulnerabilities that are found in sometimes many decades old software, since vulnerabilities are much harder to find than backdoors. For example shellshock was 30 year old code, PwnKit 12 and log4j was ~10 ish.
So if backdoors were commonplace, we probably would've found more by now.
Perhaps that's changing now, the xz backdoor will for sure attract many copycats.
Are there though? Even if true, there are probably enough places with very few eyes on them.
A kind of online-tool that collects the sources to build some relevant distributions, a web front-end to show a random piece of code (filtered by language, probability to show inreasing by less-recently/frequently/qualified viewed) to a volunteering visitor to review. The reviewer leaves a self assesment about their own skills (feed back into selection probability) and any potential findings. Tool-staff double-checks findings (so that the tool does not create too much noise) and forwards to the original authors (bugs) or elsewhere (backdoors).
A bit like wikipedias show random page.
A healthy feedback loop would have trended the average age of each vulnerability at the time of detection to be *short".
So I learned yesterday what a Trie is.
Does "think of half" apply to the folks trying to solve murders?
I got the idea from fiction (specifically Dorothy Sayers), but the number of murders Harold Shipman committed before anyone even noticed makes it plausible that people with relevant expertise (doctors, pharmacists, cops, etc.) could easily get away with murder. If Shipman had stopped after the first 100 or so he would have.
Well, at least that one is the most sophisticated one on the market (as of now) and Team Jorge is probably making shitloads of money with it while not giving a damn about who uses their software in the end.
Or most murder investigations are (by definition) incompetent.
Or (more likely): The old idiom quoted above is stupid and useless. (That it presumes that murdering and getting away with it is somehow a noble or esteemed deed should be damning enough.)
There’s no money or benefits in solving crimes. It could be done easily in many cases but nobody cares about certain people like gang members. Lots of cases where the murderer tells everyone but nobody cares.
Which part is wrong? Only 2/3 of these to choices can be wrong. The remaining one must be correct.
In both cases, the premise is unclear so good luck!
The 4th option they may appear to propose suggests that murder investigators don't get paid -- neither in money, nor in benefits.
So, to that end: As far as I know, that's not usually the case with government employees, and it is always actionable when it does happen to be the case.
In the U.S., I suspect a majority of the murders technically unsolved by police are cases where the identity of the perpetrators is somewhat of an open secret within communities that don't trust law enforcement (and LE similarly has little interest in working with them either.)
You must watch out when reading the German crime statistics. "Solved" which is marked as "aufgeklärt" in those statistics just means that a suspect has been named. Not that someone actually did it/has been sentenced for the crime.
>https://de.wikipedia.org/wiki/Aufkl%C3%A4rungsquote#Deutschl... 2nd sentence
What does that mean?
It happens loads too, frequently in high profile stuff on the news they'll have a suspect who's somehow close to it, arrest them, but then they're released once satisfied with their allibi or whatever.
I'm positive their deadline changed due to @teknoraver's patch in libsystemd.
[1] https://www.brendangregg.com/blog/2024-03-17/the-return-of-t...
Yes this was sneaky and yes this was a "slow burn" but is there really anything in the xz case that requires more than just a single competent person? Anything that requires state-level of sponsorship? The fact that random individuals online are able to dissect it and work things out suggests that it is comprehendible by a single person.
What is to say it that this was not just one smart-yet-disgruntled person acting alone?
The hack took someone with a lot of skills, determination and time.
But doesn't that describe a significant portion of open source developers?
There is also clear motivation. Wouldn't the exploit have been worth many millions on the black market?
The total value of the prize, if successful, would be worth a lot if you could sell it, but the odds of successfully getting an exploit into major infrastructure that goes undetected for long enough for your customers to buy and use it are tiny. States can afford moonshots, but I tend to expect private individuals to seek targets with a higher expected value.
Of course, that doesn't mean it was a competent state actor or that they allocated a ton of resources to it.
If you're a very skilled and dedicated hacker, what other targets do you have that can net you many millions of dollars?
> or an individual with a motive other than profit
Isn't one of the most striking things about the hacker community the extreme amounts of time and effort that are put into things that are not expected to generate any profit?
I mean, there are people who spend all their free time over several decades just digging tunnels under their property. Or build a 6502 CPU from discrete transistors. Etc.
You're conflating the hacker-as-in-threat-actor community with the hacker-as-in-Linux-maintainer community.
But in this case it is not hard to imagine that the XZ-perpetrator came from the second group, right?
Edit: I mean, this wouldn't be that different from when Ken Thompson demonstrated how to do a hidden backdoor in the C compiler?
Cite, on that? I have seen a lot of speculation from people in the know, but not this.
I'd have to assume since there's anti-debug functionality that the code is also obfuscated. Since it shipped as an opaque binary I assumed at least some of the code would be encrypted with keys we don't have (similar to parts of the STUXNET payload).
> I'd have to assume since there's anti-debug functionality that the code is also obfuscated.
Not really, as above it was done at build time.. So you have already set your home up.
It's shown the problems with package managers not taking source from the right place.
The backdoor relied on the source in the tarballs being different from the git tag, adding additional script code. This is common for projects that uses GNU autotools as build system; maintainers traditionally run autoconf so that users don't have to and ship the results in the tarballs.
I agree that this should be discouraged, and that distros should, when possible, at least verify that tarbal contents are reproducible / match git tags when importing new versions.
A lot of these tracks could be intentionally manipulated by a sophisticated actor to disguise their identity, but it's not "nothing".
Ya'll must be super fun at parties.
I'm going to stop now. You have quite effectively proven the opposite of your position.
https://rheaeve.substack.com/p/xz-backdoor-times-damned-time...
I had the same thought myself initially, but the analysis suggests a work-week that includes Fri, which precludes Israel (where the work week is Sun-Thu and not Mon-Fri), as well as celebrating Christmas and New Year's which are not official holidays in Israel. It isn't uncommon for younger people to take a day off for New Year's since it is an excuse to party, or for Jews with eastern European origins to celebrate Novy God, but I don't know of any Christmas celebrations.
Obviously these could be faked, but then why fake a Mon-Fri work week and not also fake the work hours to match it? To me it seems like an unlikely hypothesis.
I don't think that you can rule out any country based on email and commit timestamps, the attacker could have been further east and had a late work day, or further east with an early work day
(I wouldn’t put it past the intelligence services to keep working during official holidays as a form of obfuscation, mind you. I just wanted to point out that it isn’t as straightforward a deal as Christians—or atheists for that matter—existing: Israel sure makes daily life hard for them.)
Speaking of innacurate things, since the actual XZ git.repository isn't hosted on GitHub, could there be logs of which IP address the attacker was operating from?
Since it would be quite a lot of code that has been committed as well, would also be interesting to see if code style differences could be found pointing to more people involved than only one.
Not just for obvious reason of not wanting an unknown party to have RCE on your infrastructure. I think as people keep digging they will eventually formulate a payload which will allow the backdoor to be used by anyone.
As bad as it is for a single party to have access, it's much worse for any (every?) party to have access.
"In December 2015, Juniper Networks announced[55] that some revisions of their ScreenOS firmware used Dual_EC_DRBG with the suspect P and Q points, creating a backdoor in their firewall. Originally it was supposed to use a Q point chosen by Juniper which may or may not have been generated in provably safe way. Dual_EC_DRBG was then used to seed ANSI X9.17 PRNG. This would have obfuscated the Dual_EC_DRBG output thus killing the backdoor. However, a "bug" in the code exposed the raw output of the Dual_EC_DRBG, hence compromising the security of the system. This backdoor was then backdoored itself by an unknown party which changed the Q point and some test vectors.[56][57][58] Allegations that the NSA had persistent backdoor access through Juniper firewalls had already been published in 2013 by Der Spiegel.[59] The kleptographic backdoor is an example of NSA's NOBUS policy, of having security holes that only they can exploit."
Scary.
> == Observing Impact on openssh server ==
>
> With the backdoored liblzma installed, logins via ssh become a lot slower.
>
> time ssh nonexistant@...alhost
>
> before:
> nonexistant@...alhost: Permission denied (publickey).
>
> before:
> real 0m0.299s
> user 0m0.202s
> sys 0m0.006s
>
> after:
> nonexistant@...alhost: Permission denied (publickey).
>
> real 0m0.807s
> user 0m0.202s
> sys 0m0.006sIt's not hard: it's either easy, or impossible, depending on the culture at your shop.
At the basic level it's trivial - you instrument code and interpret the results against the HW and lower level machine counters.
Depending on $lang you have many good libraries for this kicking around, the hard part is working with a corp that values performance (99% do not) so none of the required mindset and surrounding infra will be setup to permit this in any useful capacity.
> but the backdoor made each login attempt use a significant amount of CPU time to check whether it was an encrypted request from the attacker
Absurdly enough, the high cpu usage is well before it even tries to figure that out. Due to avoiding "suspicious" strings in both the binary and memory, it has a more expensive string matching routine. Finding all the symbols it from the in-memory symbol tables ends up slow due to that.
That's why even sshd -h (i.e. printing help) is slow when started in the right environment. There's not enough visibility into other things that early during process startup (this happens long before main() is called), so they couldn't check what key is being presented or such. They really "should" have deferred much more of their initialization until after the ed448 check happened.
> (and this particular machine had its sshd open to the Internet because it was a cloud machine being accessed via ssh through the open Internet)
Unfortunately not. It's a machine I have at home, with some port of my public IP redirected to it, as I often need it when not at home. Oddly enough, my threat model did not include getting attacked with something as sophisticated as this, so I thought that was fine (only pubkey, no root).
Wow, that must have really sucked. It's one thing to have a rented office downtown (a VPS or similar) be backdoored, but to find a burglar broke into your home while it was under construction (or renovation in case it was a distro upgrade) and added a secret passage to it from the outside? What else did the burglar mess with while you weren't looking?
> Oddly enough, my threat model did not include getting attacked with something as sophisticated as this, so I thought that was fine (only pubkey, no root).
Most people's threat model is "everything except sshd is risky; openssh is from these cool paranoid people at OpenBSD and, as long as nobody can password guess, it's safe from non-authenticated attackers". That is, often sshd is the only open port (which helps explain why the attacker fixated on it).
(Edited to remove snark.)
If there's some piece of knowledge that's absolutely, positively critical to my life, it will exist somewhere that actually matters, not on Twitter.
I know I'm missing out on good content, and I don't care. I have _some_ self-respect.
Yes, you casual reader that keeps posting on Twitter due to laziness and momemtum, I'm absolutely talking about you. Your laziness is hurting everyone. And I'm not alone, you're limiting your audience and prioritizing, well, people too lazy to get off Twitter, and ignoring the technical, prescient (observant, at this point?), informed crowd that have left for elsehwhere. /shrug
If you’re sent a random link you can’t tell if it’s a one-off or a 45 tweet thread.
It looks open but it’s clearly not.
Oops, now that's giving:
Instance has been rate limited.
Use another instance or try again later.
How do they keep theirs up and running?
I say 110% not to be hyperbolic -- It shows you non-latest tweets on profiles, it doesn't let you see tweet threads or replies, even from the original poster when they post a chain of tweets. I literally can't read any of this content save for the threadreadapp link.
...
Making the account wasn't that annoying. Once upon a time they demanded my phone number and that was obnoxious to the point of noping out, but nowadays (after Musk took over?) they also take email instead. Email for registering accounts is nothing new, so no big deal; been doing that since 2002 when I registered my first forum account.
After I made my account I went and followed all the accounts I usually follow, and my recommendations got relevant in very short order: Posts from illustrators, the games, and players who play those games.
So, thanks Musk. You've at least convinced one guy to make an account where Dorsey flatly couldn't, and made the guy even happy about it which was pleasantly surprising.
oh yeah? Interesting you chose not to elaborate on how, given the statements I made about ruining public access stand.
But in summary, you're saying you used the platform the same way it was usable before Elon bought it, other than all of the things I mentioned that make it unusable for those not-logged-in? Let's be frank, Elon made it 200% worse, than relinquished to only 150% worse, and that's a win?
I don’t think it means currently it is reasonable , just that things are continuing to worsen and we have not yet reached a bottom even if a bottom exists.
Except not formatted as a tweet thread nor hosted on Twitter, right?
Edit : Check comments.
Yes, the backdoor hasn't been decompiled/reverse engineered yet. But it feels like clickbait to say : "It goes deeper"... Obviously. Nobody knows what it fully does yet. There was no assumption of knowing what it did.
'XZ backdoor: "It's RCE, not auth bypass, and gated/unreplayable."' [0].
[0]: https://news.ycombinator.com/item?id=39877267 (811 comments)
We will figure it out. With all of our stubborn ways, we can document it. Clap