OpenSSH Backdoors
blog.isosceles.com
blog.isosceles.com
As far as we know.
Related, there was a pretty interesting backdoor-by-bug attempt on the Linux kernel (at least, one that we know of) back in 2003: https://lwn.net/Articles/57135/
The Linux "bug" was unsophisticated by modern standards, but you could imagine a modern equivalent that's harder to spot:
Make the "bug" happen across several lines of code, especially if some of those lines are part of existing code (so don't appear in the patch being reviewed). Ensure the compiler doesn't warn about it. Make the set of triggering events very unlikely unless you know the attack. It would be very surprising to me if three letter agencies hadn't done or attempted this.
Anyways, in response to the backdoor in unrealircd, Core Security came up with a "hiding backdoors in plain sight" challenge: https://seclists.org/fulldisclosure/2010/Jul/66
"Bugdoors" are not new, and I'm sure some have been patched without anybody realizing they were introduced maliciously.
Direct network facing code is much more likely to have stringent testing and code review for all changes, so as of now it seems a bit easier to target codebases with very little support and maintenance compared to an attack that would target higher value code like OpenSSH or zlib.
Theo did not belive this and publicly disputed these claim and even revealed the name of the whistle-blower. But I have always felt the story rang true and Theo sound not have been so dismissing.
Can't find the story, but it should be on the mailing lists somewhere
Also, other articles stated that never happened.
Plus, the "backdoor" in OpenSSH was a Linux only thing possibly related to systemd. It never affected OpenBSD. That is because of Linux people patching OpenSSH with "dependency hell". I believe systemd people is doing something about these dependency chains.
Edit: the discussions and auditing of the IPsec stack happened around 2011 if memory serves me right. The supposed backdoor "happened" a decade earlier. The "agent" was named Greg Perry.
Not to mention Twitter files.
These are just the companies with enough legal gravitas and social gravity to even mention the canary dying.
Amen!
On that note: Why is it so so difficult to set up a (rootless) container/sandbox correctly? (I mean, look at what runc does – shit's incredibly complex!) And why is it next to impossible to nest containers without privileges and arcane knowledge of how some of the underlying kernel syscalls work? Even those 250 lines of code to landlock `make` that the author mentions sound awful to me.
I don't want to have to set up a sandbox for every single application by hand, let alone set up rules for all things that a malicious application could possibly exploit. Instead, I want security-by-default! Have every application run in a tight sandbox by default and let the application specify what permissions it needs, so that I only need to review those and can grant them as I like. Meanwhile, deny access to everything else!
Clearly, we are not (Linux is not) ready for this yet – we lack both a good UI for all of this permission handling and an agreed-upon contract that all application developers can follow.
In fact, for the vast majority of applications we don't even really know (have documented) what permissions / access to kernel syscalls they would need, so it'd be incredibly hard to switch to a principle of least privilege-based approach over night.
But man, one can always dream…
PS: Neither "Just rewrite everything in X where X = Rust", "just use fuzzing", or "just use MISRA coding standards" doesn't get us there. Holistic improvements help, but not with the fundamental deficiency above.
The more realistic answer is to use safer languages than C for this sort of critical work. Rust and Ada spring to mind. They could even expose a C ABI/API. From a quick search, it looks like this has already been achieved for TLS, but it sees little real usage. [0] There are also Rust implementations of SSH (Russh and Thrussh), but I don't think they expose a C ABI/API.
I'm surprised this rather obvious solution doesn't seem to even receive serious consideration. I'd be surprised if performance was seriously impaired. This blog post [1] found Rustls to be faster than OpenSSL. I couldn't find a similar performance evaluation for Russh or Thrussh.
That's not what I said, and it's not what we're talking about.
There's very little formally verified software out there, due to how challenging it is to use formal methods in large software solutions.
It makes little difference here whether the developer does it for a living. C codebases are prone to certain kinds of bugs that don't tend to arise when a safer language is used. We see a steady stream of these sorts of defects in FOSS C codebases and proprietary ones. Microsoft's closed-source TLS library has had similar issues, [0] as has Windows more broadly of course.
All that said, it would be great to see a serious and well-funded effort at a formally verified TLS implementation. Until then, we have the option of just using safer languages, but it's not getting much traction.
[0] https://signatures.juniper.net/documentation/signatures/SSL%...
TL;DR: Clever C programmers can hide very evil shit in plain sight.
[1] https://www.underhanded-c.org/ [2] https://www.ioccc.org/
I helped out reviewing the winning contribution[1] by Linus Åkesson, for what I now know was the final contest.
Linus, aka ”LFT”, is a remarkable and truly talented person in so many ways. If you haven’t heard of him before, I suggest you check out all of his projects.
[1] https://www.linusakesson.net/programming/underhanded/2015.ph...
https://news.ycombinator.com/item?id=39878681
xz/liblzma: Bash-stage Obfuscation Explained
This talk by Denzel Farmer at Columbia isn't a complete disassembly of the payload but it's the best I've seen so far.
Slides if you don't want to watch the video: https://cs4157.github.io/www/2024-1/lect/21-xz-utils.pdf
The link you want from that is this https://bsky.app/profile/filippo.abyssdomain.expert/post/3ko... ; that set of tweets has the high level overview.
They in turn links to https://github.com/amlweems/xzbot which has more details.
The TL;DR is that is hooks the RSA bits to look for an RSA cert with a public key that isn't really an RSA public key; the pubkey material contains a signed & encrypted request from the attacker, signed & encrypted with an ed448 key. If the signature checks out, system() is called, i.e., RCEaaS for the attacker.
https://lwn.net/Articles/967192/
But if there's a part that's still unclear, maybe there's another writeup somewhere that addresses it?
It's only a transitive dependency of sshd on Linux distributions that patch OpenSSH to include libsystemd which depends on xz.
It's wholy unreasonable to expect OpenSSH maintainers to vet contributors of transitive dependencies added by distribution patches that the OpenSSH maintainers clearly don't support.
Ah, ok. Then my question should really be about the distros--did any of them spot the co-maintainer being added and do due diligence?
As for the "libsystemd" part, there's another reason for me to migrate to non-systemd distros.
> Very annoying - the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it's "great new features". We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo.
> He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.
Actually just by shuffling these characters you have a good chance to get some specious translations (adding a punctuation makes it more likely to generate a completed sentence): "祪癁番䔽䔽！" -> "I am so sick!" "獶獶祪灵癁癁癁！" -> "The soul is full of blood!"
For example, 'backdoor'.encode('ascii').decode('utf_16_le') == '慢正潤牯', which Google Translate turns into "Slow and positive", but it's just nonsense.
I say ҏӲҨЏ ҜъКѠ ЇЩіН гӞэѷ in "Russian", Google Translate says "Let's talk about it".
So equivalent English gibberish would be like "hast prank bibble done anut me me ions." Google translates this one to "对我而言，恶作剧已经完成了。" (To me, the prank has been done.) in Chinese -- very valid sentence, and "¿Me has hecho una broma a mí, Bibble?" in Spanish -- also seems valid.
I guess the model is (over) optimized to generate valid outputs. This can be a feature, so it still translates grammatically invalid but to some degree understandable text (like with typos or non-standard Internet language).
Brazenly stupid and pointlessly broad: the motivation could just as easily have been to cause mass societal disruption (“terrorism”) instead of a targeted attack that just happened to sweep up everyone else in its arms.
Now, have a look at machine code which is spit out by compilers... and there, right there... no bug or backdoor in the source code though, but yet... how would you call that? compiler-injected-door?
To fight this you just need to trust and audit those small pieces of s..oftware which are gcc/clang(llvm).
CODE COVERAGE!
https://news.ycombinator.com/item?id=20383529
DonHopkins on July 8, 2019 | parent | context | favorite | on: Contributor Agreements Considered Harmful
And then there's Linus's Law, which he made up, then tried to blame on Linus.
"Given enough eyeballs, all bugs are shallow." -Eric S Raymond
"My favorite part of the "many eyes" argument is how few bugs were found by the two eyes of Eric (the originator of the statement). All the many eyes are apparently attached to a lot of hands that type lots of words about many eyes, and never actually audit code." -Theo De Raadt
https://en.wikipedia.org/wiki/Linus%27s_Law
>In Facts and Fallacies about Software Engineering, Robert Glass refers to the law as a "mantra" of the open source movement, but calls it a fallacy due to the lack of supporting evidence and because research has indicated that the rate at which additional bugs are uncovered does not scale linearly with the number of reviewers; rather, there is a small maximum number of useful reviewers, between two and four, and additional reviewers above this number uncover bugs at a much lower rate.[4] While closed-source practitioners also promote stringent, independent code analysis during a software project's development, they focus on in-depth review by a few and not primarily the number of "eyeballs".[5][6]
>Although detection of even deliberately inserted flaws[7][8] can be attributed to Raymond's claim, the persistence of the Heartbleed security bug in a critical piece of code for two years has been considered as a refutation of Raymond's dictum.[9][10][11][12] Larry Seltzer suspects that the availability of source code may cause some developers and researchers to perform less extensive tests than they would with closed source software, making it easier for bugs to remain.[12] In 2015, the Linux Foundation's executive director Jim Zemlin argued that the complexity of modern software has increased to such levels that specific resource allocation is desirable to improve its security. Regarding some of 2014's largest global open source software vulnerabilities, he says, "In these cases, the eyeballs weren't really looking".[11] Large scale experiments or peer-reviewed surveys to test how well the mantra holds in practice have not been performed.
The little experience Raymond DOES have auditing code has been a total fiasco and embarrassing failure, since his understanding of the code was incompetent and deeply tainted by his preconceived political ideology and conspiracy theories about global warming, which was his only motivation for auditing the code in the first place. His sole quest was to discredit the scientists who warned about global warming. The code he found and highlighted was actually COMMENTED OUT, and he never addressed the fact that the scientists were vindicated.
http://rationalwiki.org/wiki/Eric_S._Raymond
>During the Climategate fiasco, Raymond's ability to read other peoples' source code (or at least his honesty about it) was called into question when he was caught quote-mining analysis software written by the CRU researchers, presenting a commented-out section of source code used for analyzing counterfactuals as evidence of deliberate data manipulation. When confronted with the fact that scientists as a general rule are scrupulously honest, Raymond claimed it was a case of an "error cascade," a concept that makes sense in computer science and other places where all data goes through a single potential failure point, but in areas where outside data and multiple lines of evidence are used for verification, doesn't entirely make sense. (He was curiously silent when all the researchers involved were exonerated of scientific misconduct.)
More context:
This makes a lot of sense, because for the most part, you only go looking for bugs when you've run into a problem.
Looking for bugs you haven't run into is a lot harder (especially in complex software like OpenSSL), you might get lucky and someone sees a bug while looking for something else, but mostly things go unlooked at until they cause a problem that attracts attention.
Even when you pay for a professional audit, things can be missed; but you'll likely get better results for security with organized and focused reviews than by hoping your user base finds everything.
I think the reality is that closed source software is vulnerable to the same attack, the only difference is fewer eyes to see it and more likely a profit motive will keep those eyes directed in other ways.
But it you mention its name, you get downvoted.
Imagine a closed source company with cost pressures employing a random developer who can commit code, perhaps without any peer review, but certainly limited peer review from harried employees.
Now imagine why a nation state would want to get staff working in such a company.
Now if companies like Microsoft or Amazon or Google want to pay people to work on these open source projects that’s a different thing, and a great thing for them to do given how much they rely on the code.
Given how much closed source companies love BSD/apache/etc licenses where they can simply use these low level libraries and charge for stuff on the top I’m not sure how they would be immune from such an attack.
The risk from this was highlighted in xkcd back in 2020
If distros randomly patch OpenSSH because of SystemD, it's their problem.
- There 100% will be bad actors. Many of them.
- But not always nationstate. Instead, they do it for (dumb) personal reasons, too. Also, don't forget lulzsec as a great example of just doing it for fun. So we cannot presume to know anything about the 'why'. The bad guys I caught did it for the most asinine reasons...
But the good news is that we have options:
- Strategic: Develop processes and systems that account for the perpetual existence of unknown bad actors and allow for successful business operation even when humans are compromised.
- Reactive: Structural logging that makes sense in the context of the action. Alerts and detection systems too.
- Reduction: Reduce access to only what is needed, when it is needed.
- Proactive (not always necessary): Multi party approvals (a la code review and production flag changes or ACL changes, too)
- Social: Build a culture of security practices and awareness of bad actors. Don't make people feel guilty or accusatory, just empower them to make good design and process decisions. It's a team sport.
Bonus: By guarding against evil actors, you've also got some freebie good coverage for when an innocent employee gets compromised too!
---
Companies like Google and Amazon do the techniques above. And they don't generally rely on antiquated technology that cannot and will not change to meet the modern standards.
I know because I was the person that built and Google's first time-based access system and rational-based access systems. And multi party approval systems for access. (Fun fact: The organizational challenge is harder than the technical).
And, those strategies work. And they increase SRE resilience too!
---
But even with the best UX, the best security tooling, the best everything, etc there's no guarantees that it matters if we just reject anything except the old system we're used to.
It's like a motorcycle helmet: Only works if you use it.
A sensitive product like this would have to defend against well funded, patient, well resourced threats, including but not limited to infiltrating an organzation in order to plant code that only a few people may even be able to notice.
Corporate espionage may not be talked about very much, but it is still very fashionable. Even without state sponsored attackers.
Lots of commercial products use contractors and licensed code in the final product.
At least with most open source projects, a lot of the contribution process is in the open, so you could watch it you wanted to. As DonHopkins writes elsewhere, few people do, but it's possible. Not a lot of commercial projects offer that level of transparency into changes.
But we do know some sane things:
- The stakes couldn't be higher.
- Good: Don't allow inbound SSH connections, even through a fancy $100k firewall.
- Best: Don't let people login with SSH (treat SSH like we treat the serial port: a debugging option of last resort)
SSH is kind of a swiss army knife. But 1000x sharper ;) The delta I'm speaking of would be to have bespoke tooling for different needs. And the tooling for each purpose could have appropriate, structured logging and access controls.
With SSH you can do almost anything. But you can imagine a better tool might exist for specific high-value activities.
Case study:
Today: engineering team monitors production errors that might contain sensitive user data with SSH access and running `tail -f /var/log/apache...`.
Better: Think of how your favorite cloud provider has a dedicated log viewing tool with filtering, paging, access control, and even logs of what you looked at all built in. No SSH. Better UX. And even better security, since you know who looked at what.
---
There are times when terminal access is needed though. SSH kinda fits that use case, but lacks a lot. Including: true audit logging, permissioned access to servers, maybe even restricting some users to a rigid set of pre-ordained commands they are allowed to run. In that cases, a better built tool can allow you to still run commands, but have a great audit log, not require direct network access (mediated access) to servers or internal networks directly, flexible ACLs, and so on.
You can lock down ssh remote commands, and of course require proxied connections. You can include full auditing of command execution.
But that's either the same with any tool regardless of whether it's commercially supported / FOSS / made by anonymous devs or not. If anything, FOSS is easier to audit.
A) The ones with financial stakes in the game.
combined with:
B) Completely closed systems.
Contrarily, the most secure seem to be the ones volunteer led, with no financial stakes.
It doesn't matter what you think is true, this is clearly what is consistently happening.
- These companies underpin much of the internet's infrastructure.
- Their security practices are far more advanced than typical businesses, with SSH being a heavily restricted last resort. That's not to imply that everyone else shouldn't strive to do meet that (modern) bar too.
- Dedicated teams focus on minimizing access through time-based, role-based, and purpose-based controls.
- They actively develop new security methodologies, often closed-source, but with public evidence of their impact (e.g., https://cloud.google.com/docs/security/production-services-p... ).
- They rarely experience conventional hacks due to reduced blast radius from attacks and insider threats.
- Leading security experts in both major tech companies and niche organizations are driving new strategies and ways to think about security... their focus includes access reduction, resilience, and reliability, regardless of whether the solutions are closed or commercial for them. The ideas spread. (looking at you, Snapchat, for some odd reason)
- This is key: This evolution may not be obvious unless you actively engage with those at the forefront. I think it's what makes people think like the comment above. We cannot see everything.
- It's crucial to recognize that security is a dynamic field... with both open-source and closed-source solutions contributing.
So, the notion that volunteer-led projects are inherently more secure overlooks the significant investments in security made by major corporations that host the internet, and their relative success in doing so. Their advacements are coming to the rest of the world (eventually).
Rather than corporate lead endeavors which are very hit and miss, mostly miss, especially when the product itself claims security as a core principle.
It might not make sense to you, but the evidence points to this.
> especially when the product itself claims security as a core principle
My thought is that both volunteers and corporations contribute. In different ways, too.
One example is how a YC company made an open source version of Zanzibar. Zanzibar was an open paper to the world from Google that describes a flexible, resilient, fast access control system. It powers 99% of ACLs at Google. It's /damn/ good for the whole world and /damn good/ for developers' sanity and company security.
Corporate endeavors may fail, but they are often intense in direction and can raise the bar in terms of UX and security. Even if it's just a whitepaper, it still cannot be discounted. Besides, the larger places focusing on security aren't getting a big blast radius hack all that often, yeah?
I'm curious though, you've intrigued me. What kind of evidence or just lightweight ideas are you thinking of wrt volunteer led being more secure? No need to dig up sources if it's hard to find, but the general direction of what makes you feel that would be useful.
Historically, it's been pretty good though.
If you would consider a commercial alternative, consider how much you would need to pay to get an actionable warranty, and consider if you could find someone to do a warrantied audit of OpenSSH (or especially of OpenSSH in your environment) for that amount. It might be a better use of your money.
https://www.linkedin.com/company/teclada?trk=public_profile_...
My entire youth and professional life I've seen nothing but footguns with actual practical use of SSH. The HN community loves to hate, but the reality is that almost no one uses SSH safely. It's near impossible. Especially when it comes to configuration, keys, passwords, and network factors.
I observed the common SSH failure patterns, and I made the most obvious footguns less than lethal. Looking a step further, I made remote terminal access a pleasure to use safely even for absolute novices.
So to your point about being in YC: In doing so, I thought it would be beneficial to join a community that supports one another (YC) so that an option (Teclada) can scale to make a real impact in the world WRT the warts and footguns of SSH.
Multiplexing: This feature improves speed when using SSH to proxy applications but it also gives an authentication-free unlogged channel to phishers, allowing zero friction trivial bypass of MFA. There are ways to log this with auditd but very few companies even touch auditd much less customize it to catch phishers. With multiplexing and phishing, corporate firewalls effectively become non existent and phishing is highly effective even in places one would think it would not be. Phishing capabilities include but are not limited to any org with access to public email or public chat, Slack, Instagram, Signal, Whatsapp, Discord, etc... Loose Lips Sink Ships
Public Keys: People instinctively worry about private keys but the biggest risk in OpenSSH in my opinion is the lack of auditing of what public keys are created by whom or what and trusted on what accounts. I can for example add a key to your account if I have temporary root privs and the much later log in to a system as you do bad things and now you are the first person investigators look at. Combine this with passwordless-sudo and it's game over, not that one really needs root to pilfer secrets from a company or let competitors destroy it. In most companies this is a hot steamy mess that people choose to ignore and do mental gymnastics to avoid it all together.
How to do these things properly is a much bigger topic, too big for HN comments.
You should verify claims before mistakenly downvoting and linking to site's guidelines.
And look, it was just a simple nudge for them to disclose their affiliations more plainly in the future, while also providing relevant (and apparently appreciated) context to other readers. It was a footnote, not an argument.
Stop hallucinating like a bad LLM.
Downvoters, what are you trying to achieve?
There were some updates: https://www.zdnet.com/article/congress-asks-juniper-for-the-... and according to https://www.reuters.com/article/world/spy-agency-ducks-quest... China was behind it (but without evidence).