XZ Backdoor: Times, damned times, and scams
rheaeve.substack.com
rheaeve.substack.com
This was suggested in a different post[1], where there seem to be newly created accounts that came out of nowhere to create a sense of urgency, which might have accelerated the maintainer role transition.
[1] https://connortumbleson.com/2024/03/31/watching-xz-unfold-fr...
via: https://news.ycombinator.com/item?id=39888854
Search for "Progress will not happen until there is new maintainer" for the relevant text.
Including carefully curating things to appear in a location and to NOT appear to be more than one person.
Seems to check out to me.
In this context, middle eastern countries with Islam as the majority religion have the day off on Fridays.
Unit 8200 in the IDF especially have a very notable reputation.
That’s not to say other counties don’t have capabilities (and this doesn’t look like you need the resources of a group like say the NSA or GCHQ for this particular attack - indeed it could just be a single lone wolf) but it’s noteworthy.
But doesn't Israel also not follow a typical work week? Eg. No commits on Friday afternoon?
…What I would really want is some input from Microsoft. I don't know what they can reveal, but I believe they must have quite a bit more data, than us. Even speaking about times, well, he must have been using Github more than to just commit stuff. And I'd imagine they have some logs. Also, it's very fair to assume than he was using VPN all the time, but it's also fair to assume that he wouldn't accidentally slip like that, revealing his real time zone in 9 commits. So, yeah, for sure there are people who have waay more data than me, and probably know how to use it properly better than me too. Not sure they will be willing to share, though.
I think I saw somewhere a claim that somebody confirmed he used a VPN, I'm can't recall how they did that.
I have a relative with mental health problems who lives on the street, and he sometimes writes me from public libraries -- kind of luckily for family members who are concerned about him, the webmail service he uses still includes that info in headers. So I usually know which public library he writes from.
i.e. most Israelis work those days.
Greece and Finland aren’t prolific. I’m sure they have the resources, many lone people would have. I don’t think they have the motives though. The US, Russia, China, NK and Israel have the motive, opportunity and track record. I wouldn’t rule out Ukraine either (and Ukraine is GMT+2)
I still think that an individual is the most likely candidate though.
There are parties who have effectively endless resources and motivation to mock up a false-flag event to steer the responsibility for this a certain way. They're all very, very good at covering their tracks, and even the most experienced security researchers will take months or years to just get an educated guess of who could be responsible. Trying to figure out who did this is a waste of time.
The lesson is this:
1. Never, ever install bleeding-edge software in production, for any reason.
2. Pentesters in your organization should be regularly trying to blow holes in your stack and let both you and package maintainers know the result. If they're not, they're not doing their jobs.
3. FOSS maintainers should audit the new code in each release for security issues, particularly for things that are obviously security-sensitive.
4. Donate to your FOSS maintainers; they do insanely important work.
These rules will pay off no matter who or what wants to break into systems.
This fortunately didn't get to stable. This was as close of a shave as you can possibly get without a security Chernobyl in your systems.
EDIT:
Downvote all you want; I'm right.
There are people who very much want to convince you and everyone you know that a certain party is responsible for this and every other attack you read about. They'll go out of their way to do that. There are geopolitical goals to be achieved by doing so.
We're currently talking about banning foreign ownership of an incredibly popular web app in the US right now. If you are a government or commercial entity that would benefit from such a law, how convenient would it be if there happened to be a GitHub user with a Chinese-sounding name who committed an attack vector to the project with a timestamp that looked like it could have come out of the PRC?
And let's say it is someone in mainland China. We find out they are beyond a shadow of a doubt and who they are on a personal level. Some big-time DA's office like the Southern District of New York puts out an indictment for them and requests extradition. The people who are behind this are almost certainly state-sponsored and there's no way their asses are being handed over to US Marshals, ever. You could even argue that if they managed to stumble into a situation where they could be extradited, the government responsible for backing them would "tie up the loose end" before letting that happen, lest greater knowledge of what the organization has been doing fall into enemy hands.
Besides the Bond-style intrigue, there's no practical application to the knowledge. You already know where your users are coming from, most of the time, and might be geo-blocking based on that alone. If not, you know you should be alert to other threats from that region. If you're not, you aren't doing your job.
But for any X which is mainstream today someone was always the first of X. And even then being the second or third is still cutting edge. For a certain kind of company it makes sense to play this way, but if it were everyone then we'd have a tragedy.
99.9% of releases for packages like this are boring bugfixes and stuff, not earth-shattering new features. You're not at a competitive disadvantage for not having taken this version of xz and having routed all of your traffic through it.
Would be interesting to see a greater corpus of Jia Tan's writings, especially older ones. There are fairly distinctive idiosyncrasies that may be used to distinguish Singaporeans, Mainland Chinese and speakers of various Slavic languages (and Anglo natives).
[1] https://boehs.org/node/everything-i-know-about-the-xz-backdo...
[2] https://www.mom.gov.sg/employment-practices/public-holidays
Almost 10% of Singaporeans share the family name Tan. [1]
[1] https://en.wikipedia.org/wiki/File:Singaporean_surnames_by_f...
Singapore is an Anglophone country. Many Singaporeans are perfectly capable of writing (idiomatic) BrE or NAmE, given the medium of instruction in Singaporean education from kindergarten to university is English. Example: me.
It's not even for nefarious reasons. I travel a lot, and don't like leaking my travel itinerary on public repos.
It not only sets the timezone to UTC, but also sets the time to 00:00. So my commits will only have date information. I think only saving the date permanently in the git commit is good enough. No need to leak the precise time I made commits.
alias git='TZ=UTC0 git' alias git='TZ=UTC git'"You're in Zurich this weekend? Oh wow, me too. I'm holidaying with my wife. Let's grab lunch" as they proceed to prepare their military expense form for two business class tickets for them and their coworker
It would be like spending hours baking bread only to eat it before it goes in the oven.
But I am sure JiaT didn't only work on the xz project?
Me
> For a hacker, it is much more plausible to work in the afternoon and late at night
c/hacker/younger person
I'm 41 and have been awake before 7am maybe 20 times in the past 10 years. Half the reason I still put up with computers is because it's one of the few professions where I can work a 11-7pm schedule most days and still excel.
There are morning people and night people. You are what you are, no judgements, but it isn't something you grow in/out of, nor should be expected to.
Oh, my sweet summer child.
Talk to me again in another decade or so.
Are you saying I will grow out of this and become a lark? Or are you saying I will eventually conform, oh wise one?
https://www.sleepfoundation.org/aging-and-sleep/why-do-older...
YMMV but the idea that you can't grow in/out of sleep patterns is empirically false, and I know this from personal experience.
For most of my life, I didn't need glasses for reading... until I suddenly did. Bodies change.
The article seems to be taking about people on the verge of entering "the facility", on their slow slide into dementia, who are making do despite not getting enough sleep. I'll get back to you when I hit 70.
I don't know how in the world you got that from the article, but no.
What is this syntax? I know s/a/b/ for sed substitution but not such a thing starting with c
It's the syntax for sed substitution for someone who mistypes, is dyslexic or does not have an excellent memory.
If we're putting stereotypes aside, wouldn't we also put aside the word "hackers"?
In the xz case, the term may be accurate, but we don't know the identity of the culprit(s) and have no empirical basis for even speculating about the work schedules of such people.
> If you're having peaceful mornings, I envy you.
Well, I don't work in an office. I work alone. Like a stereotypical "hacker".
Isn't 17:27:09 +0300 == 22:27:09 +0800, making the commits just about one hour apart?
Yes, and even worse:
> Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!
No, this is a difference of almost 10 hours...
Maybe the author inadvertently switched the two analyses? Or maybe the author is just not very good with timezone math... :)
If I would be pretending something I'd certainly put at least one more indirection in.
In such setup nothing bad happens in such a leak.
- We agree it's unclear whether this was one person, or several ones. But even if it was a team, the timezone idea would still tell us where the team is located (the timings seem too much like regular office hours for a globally distributed team).
- It's also absolutely possible that commit times were changed, or commits were made/uploaded using timed scripts. Still seems like it would be hard in practice to use this to cover up a massive difference in timezone, though, because it would require adding a massive latency -- and xz wasn't developed in a vacuum, but in reaction to other online events like the filing of issues or mailing list posts.
- yes, we did get the two examples mixed up (the two times we say are 10-ish hours apart are actually only minutes apart, and the times we say are minutes apart, are 10-ish hours apart). Update is on the way.
- and finally, yes, UTC+2/3 goes beyond just Eastern Europe and includes part of the Middle East. We also clarified this in an update to the article.
DST varies by state in Australia. Western Australia (UTC+8) does not observe DST.
~~It has surprisingly many commits on Sundays.~~
UPD. Weekday 0 is actually Monday, not Sunday.
> Notably, on Tuesday, Jun 27, 2023, we see two commits: 23:38:32 +0800 and 17:27:09 +0300. If we do the math, there is about a 9-hour difference between the two commits.
Putting it all into UTC we get:
23:38 - 8 = 15:38
17:27 - 3 = 14:27
> Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!
Putting it all into UTC we get:
21:53 - 3 = 18:53
17:00 - 8 = 09:00
21:53 - 8 = 13:53
17:00 - 3 = 14:00
Are they whiteboarding the best strategy for distributing the timestamps of commits? Mandatory performance evaluations on all RCEs? Distribute the poisoned commits amongst more authors?
Jia Tan probably used a vpn though - we know that they did for accessing IRC (source: https://boehs.org/node/everything-i-know-about-the-xz-backdo...)
The Witopia VPN that he used for IRC [1] is US based: https://www.personalvpn.com/contact-us/ and they don't mention neither LN payments nor not keeping logs.
1. "~jiatan@185.128.24.163" https://boehs.org/node/everything-i-know-about-the-xz-backdo...
I don't know why most people assume that hackers even bother with stolen credit cards in the first place. I mean, they sure do, but those are your average Joes in the business of refund reshipping and other types of scams.
Those who want the maximum anonymity don't even bother with buying anything. It's as simple as going to one of the popular websites who leak databases, setting up OpenBullet software or spending anywhere from 1 to 5 hours writing custom mail:pass validators to spam requests to either API or login form through (once again) leaked proxies, etc. using leaked credentials. Or simply going into one of those threads titled 'x100 Mullvad accounts" which have already validated accounts with anywhere from 1m pre-paid to multiple years. And there's even a bonus of not being shown as a user of this account if you do not use official App and simply load configuration manually through ovpn, etc.
And then there's proxy-chaining if you're doing something truly nefarious. It's super easy to chain multiple VPNs with few socks proxies.
People behind XZ backdoor to me look much more smarter than myself, so I would bet they took care of this angle and will be untraceable.
I've occasionally sent packages postmarked as being from one zipcode from another; as long as it's in the same region, much of the postal processing doesn't care so much.
There's also remailers and forwarders.
This is why AMZN packages have a return address in Vegas or similar sometimes.
Or that they store something that allows correlating an IP+time to a Mullvad account.
When a VPN provider isn't cooperative, metadata can be sought upstream.
Mullvad deserves much credit for accepting non-digital payments, which increases the cost of deanonymization.
It’s quite plausible that they didn’t manage perfect vpn usage every single time.
The thing is you only need to make a mistake once.
It is interesting, at least one of the things listed as suspicious is something I do with some frequency. I will sometimes work on what is logically three commits and not chunk them out until the very end.
A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?
I don't think much speculation is needed. RPM and deb catches every Enterprise Linux distribution, to the best of my knowledge.
rpm catches Red Hat and all derivatives like CentOS, Alma Linux etc, as well as SuSE, Amazon Linux, and even Microsoft's Mariner Linux (now renamed Azure Linux).
deb catches Debian and Ubuntu.
That's a huge global install base just through those two targets.
And to me it's less that they don't care, more that their script probably only works or only has been tested on the ones using .rpm/.deb.
My understanding was that this was the result of a bug? But I understand that part less than any of it, so I well could be wrong.
>And to me it's less that they don't care, more that their script probably only works or only has been tested on the ones using .rpm/.deb.
Now that, strikes me as more likely. Maybe they wanted to be relatively sure that they wouldn't be detected and needed to select a test matrix to test against (SELinux/landlock config across rpm/deb distros, versus across every distro.)
I guess what I'm getting at is... did they have an intended set of targets and knew they were running deb/rpm, on top of any other factors?
The payload only works on systems that connect their sshd to systemd-notify -- which is not true for many Linux distributions beyond Debian-based and RH-based anyway (e.g. Arch Linux doesn't do this).
Everything you are saying is so far out of the realm of possibility that it makes me wonder if you follow this stuff at all.
If you had 2+ years of spare time, and these type of skills, the attacker would be far better off just trying to get a job inside the organization they are targeting.
Who says they don't? :-)
I understand that people should read the full article instead of drawing any conclusion from a mere image, but lots of people don't (hence why clickbait works). I also think it's a little tasteless, if not misleading.
Disclaimer: I'm a Chinese.
the rest is mostly speculation about where else they could have been from given holidays and times
Just a heads up for the next guy...
People who don’t read the article and find themselves lacking an informed or nuanced understanding of the contents of said article are the only ones to blame here. Full stop.
That HN doesn’t have a culture of shaming those who don’t read the article is partly the fault of the rules here which finds “read the article please” to be a distraction/nuisance but don’t mind that this often results in misinformed discussion propagating toward the top of a thread.
Someone with access to the right machines could easily check whether this is the case. A lot of projects have set up webhooks on github to sync new commits with their own bug tracker, for example, or testing infrastructure. Logs on these systems would tell you exactly when the commits were pushed.
This is an interesting take. Most software engineers I know love to show off their FOSS contributions. It's a place to do what _you_ want to do, free of budgets and MBA interference. I guess in certain parts of the world you probably want to keep that sort of skill set under wraps, though.
Simplicity and clarity translate into security, maintainability, and reliability.
Everything else: Very interesting!
Also: analysis on the email chains they interacted with. It's a possibility that they may have responded to their own emails in their mailing lists with fix suggestions/requests for clarification etc.
Edit: it’s a meme. Apparently. I feel like an old man.
> illia-git This user is suspected to be a part of an online ransomware organization. Please report any suspicious activity to GitHub security. Poland
https://knowyourmeme.com/memes/discord-user-is-a-suspected-t...
There was WAY more than one person involved here. Every email, commit, etc. was a deliberate task that was carefully planned.
Edit: speaking of, these guys might also have some time pressure at the moment with regards to hacking stuff lol. Kind of fits the bill with what we know about the hack
Start here.
That’s a stretch. Stuxnet was the first acknowledged state cyber attack, utilized multiple zero days, and destroyed nuclear weapons manufacturing facilities. Bigger in scope sure, but bigger unconditionally? I don’t know about that.
Stuxnet: aimed to delay one nuclear facility that was still being built
SSH pre-auth RCE: root access to most servers on the planet, impacting everyone from hobbyist self-hosters (that's me) to large businesses (of every type imaginable) to probably even some of the security agencies around the world. With SSH's track record, a lot of people choose to trust it as their internet-facing access protocol
I expect that whoever made this would have picked their targets carefully to avoid revealing the backdoor, so probably they'd aim (at least at first) at a handful of businesses of interest (advanced chip manufacturing, say) and governments, rather than causing tangible widespread issues in some way. Theoretically, though, (perhaps upon being discovered) if they'd just drop `rm -rf && poweroff` on all reachable systems (perhaps having it spread into networks in a worm-like fashion, setting a timer for 30 seconds so that it can propagate), most computer-based systems would just stop working (either killed themselves, or from failing routers or other systems they relied on), and lots of them would lose data because their backups either weren't working or were also impacted. Consider just how much involves a computer today, from infrastructure to hospitals. That's a whole lot more impact than a delayed nuclear program due to some unpublished Windows exploits
What modulates this potential impact is the question of how many important systems run something other than the OSes they were currently in the process of targeting (Debian, Ubuntu, and Fedora afaik), how many servers require a VPN (or similar) to access ssh and have no outward-facing ssh server anywhere on their network, and how many years it would have taken to uncover (more and more systems would have been running this over time)
No, absolutely not; that didn't actually happen. Most servers on the planet that are running x86_64 Linux are probably not running a rolling-release distro based on .deb or .rpm. Rolling-release (or a beta version of the next OS release) is required because I can't imagine one of them updating to a new version of a library like this until the next major release of their OS, and .deb/.rpm because those are the only build environments where the backdoor would get built into the library. (And on top of that, only systems where sshd is patched to link to libsystemd.)
Also: while I will admit that there are many businesses with abysmal security practices, most will not have ssh exposed to the public internet; that'll require access to the corporate VPN as well. You note this, but fail to recognize the impact.
Even hobbyists probably aren't hit by this in large numbers. I run a VPS for a variety of things, and it runs Debian stable. Assuming this backdoor attempt was never found, it wouldn't get installed on it until mid-/late-2025, when Debian trixie is likely to be released (plus some time for me to get around to upgrading it).
I did have the affected package on my laptop, which runs Debian testing. But I don't have sshd enabled on it all the time, and even if I did, my laptop is rarely on a network without NAT (and I usually opt to tether to my phone when in public rather than use public WiFi). All the other machines on my home network are also running Debian stable, and were unaffected.
I suspect that precious few systems even had the backdoor installed on them, and of those, even fewer were accessible directly on the public internet.
Now this could have been bigger than Stuxnet, if the backdoor had remained secret for -- and I think I'm being generous here -- another year or so.
See modulating factors at the bottom. In this case, it didn't because it was caught before making it into a stable release, but only due to sheer luck. The story is still so much bigger than Stuxnet to me because its aim was an actually tangible impact on millions of people's lives
> Now this could have been bigger than Stuxnet, if the backdoor had remained secret for -- and I think I'm being generous here -- another year or so.
Agreed on that! (To be clear, I still think it's already bigger, not because of the impact it concretely had but because I perceive it to having been very close; but I can understand this/your point of view very well)
Or that, once in the wild, that the author would have their pick of the litter when choosing targets. It's also true that the exploit has several kill switches in it, and so even vulnerable servers could be protected by simply setting an environment variable.
Which, honestly, makes this entire attack all the more strange. It had incredibly unclear value and longevity. It makes me wonder if the author didn't have different, parallel aims, or if this wasn't a type of practice run for a larger attack in the future.
Sure, if the backdoor hadn't been found for a couple more years, the impact would have been much higher, as the backdoored version would have made it into actual current releases of the popular server distros, and companies gradually upgraded.