But yes, some black hat had a really really bad day: being so close to owning any ssh-enabled recently-updated box and when they can finally see a finish line it is just poof... gone in a day. It was two years of effort from at least one guy and more likely a team.
The honorable thing for "Jia" to do after this epic failure is seppuku, or whatever his or her local equivalent is.
People like "Jia" are pure parasites. It's one of the best cases of "why we can't have nice things" I've ever seen.
It would definitely be interesting to see what would happen if the attack wasn't noticed, but instead people focus their interest on attacking Jia Tan because "wow, that guy is one hell of an asshole, it sure is a great thing that he failed!".
Whether or not this attack was the rare one that failed out of many similar ones is largely irrelevant to people. Quick, discuss this one particular case where we noticed it, news flash and all.
> People like "Jia" are pure parasites
They are "parasites" because they don't do what they are "supposed to"? That's pretty crazy. I guess every person that's doing what matches their interests is somehow a bad person/parasite/etc. Or is that only if what they do is forbidden by some rule book? Do you see what I'm getting at here?
"pls sir dont create linux backdoar"
If they inflict harm on their fellow humans in order to do so, then literally yes.
There's a reason why a society will collapse if you have too many of this type of person.
What I'm saying is that there are a lot more technically legal ways to profit that harm society, some of them more nefarious than what Jia Tan did.
Doing things that are bad for the society in a fucked up society seems justifiable. It doesn't necessarily make you a bad person.
People just have a more averse reaction to things that are obviously bad, even if in practice there are way worse things that initially seem innocuous and are actually legal to do. That's just the textbook example of hypocrisy.
The fact that many projects are now searching deep and wide through their commit histories, looking at just who they're taking code from, beginning to develop frameworks for attack mitigation and remediation... an entire type of previously very-promising attack is completely burned. This has been a massive defeat. And it happened all by chance.
I fear it's not just the attack that is burned. If new contributors have to be distrusted and/or go through some sort of vetting that isn't based on the merit of their contributions, that is a terrible blow to the entire open source movement.
The threshold for young coders without a great deal of history and without a network of contacts to become contributors to important open source projects has just gone up massively.
The bar is still trivial. The worst part is that it will be difficult to be anonymous. There are a lot of valid, even life & death reasons for anonymity, and that will now be much harder.
So the loss is all the contributors who don't dare let their government or employer see them helping the wrong projects, or in some cases don't want to disclose that they are any good at coding or particularly interested in security etc.
I would expect to see more projects try to protect against this in general by attempting to separate the building and testing procedures.
No, it was not.
The test files were used as carriers for the bulk of the malicious code, but running the tests did nothing malicious; the supposedly corrupted test file did act as a corrupted file in the test. What extracted and executed the code hidden in these test files was a bit of code injected in the configuration steps, which in turn modified the compilation steps to extract and execute more code hidden in these test files, which in turn extracted and injected the backdoor code. This all happened even if the tests were never run.
> I would expect to see more projects try to protect against this in general by attempting to separate the building and testing procedures.
The suggestions I've seen on this line of thinking were to not only separate the building and testing steps (so that the testing steps cannot affect the output of the build step), but also to remove all test files before doing the build, and adding them back for the testing step. The important part being to remove all binary test files before doing the build, and having two separate steps only as a side effect of not being able to test without putting the test files back.
I don't think this approach is going to be adopted, as it's too much work for little gain; people are focusing more on making sure all files on the tarball used for the build either come from the corresponding tag in the repository, or can be proven to have been deterministically generated from files found in the corresponding tag in the repository. That would have either caught or ignored the key file added to the tarball which started the extraction and injection steps.
Just make the two separate projects, where the test project depends on installed binaries.
The government who paid them just lost the investment, but who cares about them? It's only a good thing if this does not get a reputation for being a good invesment.
If it was a criminal, then the same as the government they lost the investment but again that's only a good thing.
The statement was that someone had a really bad day.
The implication that people are driven by money and nothing else, or even that they have no right to feel like they had a bad day if they were paid, is absurd.
Nobody is saying that you should be sympathetic. It’s just an interesting comment: An interesting thing to think about. A worthwhile contribution to the conversation. This was someone’s very bad ending to a very long project.
> If so, then they were paid for those hours and are perfectly whole right now.
People are not ideal money-fed automatons. Even if you're fully paid for your software development, it feels bad to see all your work thrown away.
> maybe their career takes a dent but I do not weep for anyone who worked on this
Even if they were fully paid, and even if their career is not affected at all (or even improves, "I wrote the xz backdoor" would be an impressive line in a curriculum if you disregard the moral aspects), it can still feel bad to see your work thrown away so close to the finish line. People are not career-driven automatons.
But I agree with you, I do not feel bad for the author of this backdoor; whoever did that does deserve to see this work unceremoniously dumped into the trash. But I can understand why the ones who worked on that code would feel bad about it.
I don't see why; news coverage has pretty uniformly taken the view that "this guy was an evil genius, and we escaped his incredibly well-executed plot by the skin of our teeth".
Assuming they work 40 hours a week and are doing this in a team (presumably every major player has such a team or is scrambling to get one now), one must expect many potentially infiltrated projects out there.
I'm not familiar with any programming language that provides tries in a standardized way. They're not so hard to code, so I wonder if this will become a trend in future malware.
Aside, this is one of my grumpy-old-dude opinions when it comes to regular non-malicious code: Source "greppability" is a virtue, one that must be balanced against "don't repeat yourself" and "compile time checking", etc.
Some examples off the top of my head:
1. One import/alias per line, no wildcards.
2. If you have to concatenate a literal string across multiple lines, try to ensure the break isn't inside an interesting substring someone would search for.
3. If you write a compile-time constant like foo=4*6*9, the comments around it should also contain the result. Sure, someone might screw up and forget to keep the comment in-sync, but it can also be invaluable when someone else is tearing their hair out trying to find out why the production logs are saying nothing but "Illegal frump: 216".
Don't get me wrong, I love a good IDE, and many will offer "Find Usages" or "Find Literal"... but it doesn't always work all the time, for all languages in the project at once, for everyone on a team who might use other tools, or during a PR review through a web-browser, etc.
This is called semantic line breaks and is also important to get (admittedly more naive) diffs sensible.
You think these guys have standups?
https://rheaeve.substack.com/p/xz-backdoor-times-damned-time...
There are other countries on the same time zone, but only Israel has history of using and selling hacks of this level.
Or someone further east with a classical hacker "let's get up late in the afternoon, and start the serious hacking late at night."
I think you're missing where the difficulty is. I'd argue the technical knowledge here isn't the hard part here. You can find more than enough folks who would know how to do everything you listed outside of any state-sponsored context. The hard part is the methodical execution, including the opsec, backup plans, plausible deniability, false identities, knowing how to play the long game, being able to precisely target what you want, understanding human psychological tendencies, identifying unforeseen security vulnerabilities, etc... these are the things that I imagine are likely to distinguish talented individuals from state-sponsored actors, not something like "knows tries and x86 assembly".
And taking all those into account, this absolutely smells state-sponsored. I just see no way that a private entity had enough of an incentive, foresight, experience, and patience to play such a long game with such careful maneuvers.
The assault was comprehensive, holistic and systematic in its approach – this article does not mention it, but other reports have indicated that the person behind it also managed to compromise the PKI layer at the edge between OpenSSL and sshd which brings an extra level of complexity to the backdoor.
Doesn't this apply to many OSS contributors? people with skills and free time?
Criminals with the resources to maintain such a complex operation for two years?
I also am a little biased and assume being burned in state-sponsored acts is similar to the no-blame culture of breaking infrastructure in tech :) because by all accounts this compromise was extremely well done, until it wasn't.
Also, we can't be sure the compromise wasn't intentionally telegraphed to cause some other action (using a different library) on purpose.
That actually makes me think it's not happening at a larger scale, since we'd likely have heard of at least a few similarly elaborate cases being uncovered by now. If not during the attempt itself, then at least at some later point in time.
Either almost all of these operations remain undetected, because they are even more sophisticated and much of the world's software ecosystem has been secretly compromised for years or there aren't actually that many such operations.
It's one thing to attack a target, it's quite another to try to give yourself a master key to every rpm or deb linux box in the world.
Quite confusing, because the two different meanings are nearly opposite to one another.
See also: https://www.merriam-webster.com/grammar/nonplussed
It's pretty easy to see where the innovative sense came from: "plussed" doesn't mean anything, but "non" is clearly negative. So when you encounter the word, you can tell that it describes (a) a reaction in which (b) something doesn't happen. So everyone independently guesses that it means failing to have much of a reaction, and when everyone thinks a word means something, then it does mean that thing.
You see the same thing happen with "inflammable", where everyone is aware that "in" means "not" and "flame" means "fire". (Except that in the original sense, in is an intensifying prefix rather than a negative prefix. This doesn't occur in many other English words, although "inflammation" and "inflamed" aren't rare. Maybe "infatuate".)
Once the first stage, extracting a shell script from one of the „test“ data blobs, has been found it was clear to everybody that something fishy is going on.
It’s inconceivable that I’ve would have found the first stage and just given up, but then it was „only“ a matter of tedious shell reversing…
They could easily done without the „striping“ or „awk RC4“, but that must have complicated their internal testing and development quite a bit.
For a one of its kind deployment it would probably not matter. However, deploying to multiple targets using the same basic approach would allow all of them to be found once one was discovered. With some mildly confsing but different scripting for each target systematic detection of others becomes more difficult.
But what you were looking at might not be the first stage.
You might be looking at the modified Makefile. You might be looking at the object files generated during the build. You might be looking at the build logs. You might be investigating a linking failure. The reason for so many layers of obfuscation, is that the attacker had no idea at which layer the good guys would start looking; at each point, they tried to hide in the noise of the corresponding build system step.
In the end, this was caught not at the build steps, but at the runtime injection steps; in a bit of poetic justice, all this obfuscation work caused so much slowdown that the obfuscation itself made it more visible. As tvtropes would say, this was a "Revealing Cover-Up" (https://tvtropes.org/pmwiki/pmwiki.php/Main/RevealingCoverup) (warning: tvtropes can be addictive)
Because anyone has the resources to test this locally and find the issues, but they just didn't have the time anymore.
Or was xz only one of multiple targets where the overall plan didn't actually fail and an updated backdoor, without the performance issues, will soon be injected again through some other project.
Surely that would be something the current ais could do 100x better?
So I wonder if there isn't a business there in scanning anonymous posts to associate them with others to deanonymize them all?
You could obviously use an ai to rewrite your text, but I bet there is something there that can still be correlated. Simply avoiding using favorite words and phrases doesn't change the essence of whatever you're saying.
I think the concept of sockpuppet is about to get complicated. Really it probably already is. If I can imagine something, someone else has probably already been doing it and countering it, and countering that for a decade.
https://theconversation.com/how-the-unabombers-unique-lingui....
Jia Tan mook: "Sir, they found us out."
Jia Tan officer: "What's the sitrep, are they coming for us? Are they getting our operatives in the field?"
Jia Tan mook: "No sir, they're just reverting and blacklisting our accounts and releases. Muttering something about 'uptime'."
Jia Tan officer: "Shit, we're dealing with sysadmins."