Hacker News Reader

Comment by coldpie | Hacker News Reader

This is IMO one of the coolest tech stories to ever happen, seriously amazing spycraft & hacking skills, but I haven't been keeping up with new developments from this story since it broke. Last I heard, the best guess at what happened was some state-sponsored actor worked very hard to get this merged, and it was caught luckily at the last minute. But no one had any smoking gun as to who did it or why or who they were targeting. Any new developments since then? Are we still just totally in the dark about what was going on here?

View on HN

> and it was caught luckily at the last minute

This isn't correct at all. The changes were merged into xz and made it into testing branches of major Linux distros.

It was caught at T plus a few minutes only because a neurotic Microsoft employee performing debugging noticed an obscure performance issue.

You can literally say Microsoft saved Linux that day. Imagine thinking this 25 years ago.

It's the difference between something really bad which happened, and something really, really, really, really bad: a malicious actor having RCE credentials to every new Debian and Red Hat box on planet Earth.

Redhat actually stumbled on the bug separately with valgrind errors triggering, so it's days were likely numbered regardless. Probably saved them a lot of debugging but the writing was on the wall.

A lot of people fail to fully grasp how bad this could have been on the off chance the authors were slightly less sloppy.

Imo it just proves there's a 99% chance a standard distro has a current zero day in it.

If a state actor (it almost has to be a state actor at the time frame they were operating under) could put in this much effort once, they clearly could afford to do it X times. And when you look through the history of communications from the author, it just reads like 'another day at the office'.

Red Hat noticed that something was off, but there was a new version published by "Jia Tan" that fixed the warnings and the performance issue, so it's not really clear that the original version would have still gotten as deep of an investigation as would have been needed to find the issue.

It's possible though. The noise around it did at least put Freund on alert and we should be very glad both that "Jia Tan" made the mistakes they made originally and that Freund followed up on their gut feeling

> Red Hat noticed that something was off, but there was a new version published by "Jia Tan" that fixed the warnings and the performance issue

Video of Jia Tan fixing the valgrind bugs: https://www.youtube.com/watch?v=A16YuzuKN58&t=138s

The irony being that 'Jia Tan' went out of their way to ensure the backdoor was very well obfuscated, to the point it inadvertently caused bugs and slight, but noticeable, performance issues.

One wonders whether the xz backdoor would have been discovered if slightly less obfuscation was used.

The whole xz incident is a pretty strong argument to:

a) change practice from including binary (opaque) test files themselves to human-readable scripts and tooling that build test files on-demand,

b) raise suspicion of any binaries included in open source projects, and

c) create much more scrutiny around dependencies of 'highly scrutinised' packages like OpenSSH.

It's a shame that there isn't a foundation (that I'm aware of) that can donate time and effort of vetted developers to foundational open source projects like xz.

> create much more scrutiny around dependencies of 'highly scrutinised' packages like OpenSSH.

But xz is not a dependency of upstream OpenSSH you see. It was a dependency of a patch created by Linux distros for systemd integration.

The problem is that there are many many people that are falling over themselves to believe bogus claims about false positives.

Outside of Valgrind bugzilla bug reports these claims almost never stand up to close scrutiny. Not that the people making the claims ever perform any scrutiny. It's usually "my application doesn't crash so it must be a false positive" or "I'm sure that I initialised that variable" or "it's not really a leak, the OS will reclaim the memory".

Still no smoking gun, but possibly Russia. From the video https://youtu.be/aoag03mSuXQ?t=2883:

> A lot of the aliases, like Jia Tan, they sound like Asian names, and the published changes are all timestamped in UTC+8, Beijing time. So the signs point to China. And that's why it's probably not China. I mean, why would they make it that obvious? Every other part of the operation has been so meticulous, so cautious.

> And they also worked on Chinese New Year, but not on Christmas. And over the years, there were nine changes that fall outside of the Beijing time into UTC+2, which is a time zone that includes Israel and parts of Western Russia. That's why some experts have speculated that this could be the work of APT29, a Russian-state-backed hacker group also known as Cozy Bear. But again, do we know? No, of course we don't know who it is, and we likely will never know.

Russians don't celebrate Christmas on the 25th.

That was also what I took away when watching the video. Russians don't celebrate Christmas on the 25th (they Celebrate on January 7th), but even more than that: Russians don't celebrate Christmas the same way we do in the west.

Their "Christmas" family celebrations are on New Years Eve.

So if you're drawing conclusions from them not working on the 25th (which is a literal normal day in eastern europe) then signs point elsewhere unfortunately.

ah, Eastern Europe is a homogeneous slop, good to know. The 25th is only celebrated as a national holiday in like 90% of Eastern European countries. Russia and Serbia celebrate it on January 7th instead, and even that is only due to their use of the old Julian calendar. But sure, normal day in a very narrow and obscure definition of Eastern Europe.

Calm your tits, it’s pretty clear I meant slavic nations, and yes: even the January 7th stuff is not celebrated like we do in the west.

I’m trying to convey to people why its materially a red-herring that “russians” wouldn’t work on the 25th: since its a normal day.

Those anecdotes don’t mean anything. If I were China and wanted plausible deniability I would work on CNY and take off on foreign holidays. Of course that leaves Beijing time as a weird oversight though it’s always Beijing time anywhere in China.

One can also make commits with any arbitrary date/time(zone) values they want that don't reflect reality.

>And that's why it's probably not China. I mean, why would they make it that obvious?

That's just what they want you to think!

UTC+2 isn't very convincing as an argument for Russia. Only the Kaliningrad exclave uses that timezone, and if I were in a state-backed group, I'd live in one of the big cities.

Also quick search suggested UTC+3 was seen during the summer, and Russia doesn't do DST either.

Edit: some of the UTC+2/3 times are attributable to being differences in git committer and author dates (e.g. email patches)

I’ve always found this an amusing method of attribution considering top tier hackers are unlikely to be writing code only during office hours.

I couldn't let this be, so I went through the commits and as far as I can tell, that's the case. The committer/author names and timestamps are consistent with using --author on a commit (... or in a few cases, --amend --author).

Except one: commit 3d1fdddf9 has Jia Tan as both author and committer but the author timestamp is in +0300 while the commit timestamp is +0800.

Stuxnet is also another mindblowing case. Wired write-up on it is a recommended reading: https://web.archive.org/web/20141028182107/http://www.wired....

Reply on news.ycombinator.com