Even with the best intentions, can a volunteer-driven project like OpenSSH truly guarantee the same level of security as a commercial solution with dedicated resources and a financial stake in preventing backdoors?
Imagine a closed source company with cost pressures employing a random developer who can commit code, perhaps without any peer review, but certainly limited peer review from harried employees.
Now imagine why a nation state would want to get staff working in such a company.
Now if companies like Microsoft or Amazon or Google want to pay people to work on these open source projects that’s a different thing, and a great thing for them to do given how much they rely on the code.
Given how much closed source companies love BSD/apache/etc licenses where they can simply use these low level libraries and charge for stuff on the top I’m not sure how they would be immune from such an attack.
The risk from this was highlighted in xkcd back in 2020
If distros randomly patch OpenSSH because of SystemD, it's their problem.
- There 100% will be bad actors. Many of them.
- But not always nationstate. Instead, they do it for (dumb) personal reasons, too. Also, don't forget lulzsec as a great example of just doing it for fun. So we cannot presume to know anything about the 'why'. The bad guys I caught did it for the most asinine reasons...
But the good news is that we have options:
- Strategic: Develop processes and systems that account for the perpetual existence of unknown bad actors and allow for successful business operation even when humans are compromised.
- Reactive: Structural logging that makes sense in the context of the action. Alerts and detection systems too.
- Reduction: Reduce access to only what is needed, when it is needed.
- Proactive (not always necessary): Multi party approvals (a la code review and production flag changes or ACL changes, too)
- Social: Build a culture of security practices and awareness of bad actors. Don't make people feel guilty or accusatory, just empower them to make good design and process decisions. It's a team sport.
Bonus: By guarding against evil actors, you've also got some freebie good coverage for when an innocent employee gets compromised too!
---
Companies like Google and Amazon do the techniques above. And they don't generally rely on antiquated technology that cannot and will not change to meet the modern standards.
I know because I was the person that built and Google's first time-based access system and rational-based access systems. And multi party approval systems for access. (Fun fact: The organizational challenge is harder than the technical).
And, those strategies work. And they increase SRE resilience too!
---
But even with the best UX, the best security tooling, the best everything, etc there's no guarantees that it matters if we just reject anything except the old system we're used to.
It's like a motorcycle helmet: Only works if you use it.
A) The ones with financial stakes in the game.
combined with:
B) Completely closed systems.
Contrarily, the most secure seem to be the ones volunteer led, with no financial stakes.
It doesn't matter what you think is true, this is clearly what is consistently happening.
- These companies underpin much of the internet's infrastructure.
- Their security practices are far more advanced than typical businesses, with SSH being a heavily restricted last resort. That's not to imply that everyone else shouldn't strive to do meet that (modern) bar too.
- Dedicated teams focus on minimizing access through time-based, role-based, and purpose-based controls.
- They actively develop new security methodologies, often closed-source, but with public evidence of their impact (e.g., https://cloud.google.com/docs/security/production-services-p... ).
- They rarely experience conventional hacks due to reduced blast radius from attacks and insider threats.
- Leading security experts in both major tech companies and niche organizations are driving new strategies and ways to think about security... their focus includes access reduction, resilience, and reliability, regardless of whether the solutions are closed or commercial for them. The ideas spread. (looking at you, Snapchat, for some odd reason)
- This is key: This evolution may not be obvious unless you actively engage with those at the forefront. I think it's what makes people think like the comment above. We cannot see everything.
- It's crucial to recognize that security is a dynamic field... with both open-source and closed-source solutions contributing.
So, the notion that volunteer-led projects are inherently more secure overlooks the significant investments in security made by major corporations that host the internet, and their relative success in doing so. Their advacements are coming to the rest of the world (eventually).
Rather than corporate lead endeavors which are very hit and miss, mostly miss, especially when the product itself claims security as a core principle.
It might not make sense to you, but the evidence points to this.
> especially when the product itself claims security as a core principle
My thought is that both volunteers and corporations contribute. In different ways, too.
One example is how a YC company made an open source version of Zanzibar. Zanzibar was an open paper to the world from Google that describes a flexible, resilient, fast access control system. It powers 99% of ACLs at Google. It's /damn/ good for the whole world and /damn good/ for developers' sanity and company security.
Corporate endeavors may fail, but they are often intense in direction and can raise the bar in terms of UX and security. Even if it's just a whitepaper, it still cannot be discounted. Besides, the larger places focusing on security aren't getting a big blast radius hack all that often, yeah?
I'm curious though, you've intrigued me. What kind of evidence or just lightweight ideas are you thinking of wrt volunteer led being more secure? No need to dig up sources if it's hard to find, but the general direction of what makes you feel that would be useful.
There were some updates: https://www.zdnet.com/article/congress-asks-juniper-for-the-... and according to https://www.reuters.com/article/world/spy-agency-ducks-quest... China was behind it (but without evidence).
https://www.linkedin.com/company/teclada?trk=public_profile_...
My entire youth and professional life I've seen nothing but footguns with actual practical use of SSH. The HN community loves to hate, but the reality is that almost no one uses SSH safely. It's near impossible. Especially when it comes to configuration, keys, passwords, and network factors.
I observed the common SSH failure patterns, and I made the most obvious footguns less than lethal. Looking a step further, I made remote terminal access a pleasure to use safely even for absolute novices.
So to your point about being in YC: In doing so, I thought it would be beneficial to join a community that supports one another (YC) so that an option (Teclada) can scale to make a real impact in the world WRT the warts and footguns of SSH.
Multiplexing: This feature improves speed when using SSH to proxy applications but it also gives an authentication-free unlogged channel to phishers, allowing zero friction trivial bypass of MFA. There are ways to log this with auditd but very few companies even touch auditd much less customize it to catch phishers. With multiplexing and phishing, corporate firewalls effectively become non existent and phishing is highly effective even in places one would think it would not be. Phishing capabilities include but are not limited to any org with access to public email or public chat, Slack, Instagram, Signal, Whatsapp, Discord, etc... Loose Lips Sink Ships
Public Keys: People instinctively worry about private keys but the biggest risk in OpenSSH in my opinion is the lack of auditing of what public keys are created by whom or what and trusted on what accounts. I can for example add a key to your account if I have temporary root privs and the much later log in to a system as you do bad things and now you are the first person investigators look at. Combine this with passwordless-sudo and it's game over, not that one really needs root to pilfer secrets from a company or let competitors destroy it. In most companies this is a hot steamy mess that people choose to ignore and do mental gymnastics to avoid it all together.
How to do these things properly is a much bigger topic, too big for HN comments.
You should verify claims before mistakenly downvoting and linking to site's guidelines.
And look, it was just a simple nudge for them to disclose their affiliations more plainly in the future, while also providing relevant (and apparently appreciated) context to other readers. It was a footnote, not an argument.
Stop hallucinating like a bad LLM.
A sensitive product like this would have to defend against well funded, patient, well resourced threats, including but not limited to infiltrating an organzation in order to plant code that only a few people may even be able to notice.
Corporate espionage may not be talked about very much, but it is still very fashionable. Even without state sponsored attackers.
Lots of commercial products use contractors and licensed code in the final product.
At least with most open source projects, a lot of the contribution process is in the open, so you could watch it you wanted to. As DonHopkins writes elsewhere, few people do, but it's possible. Not a lot of commercial projects offer that level of transparency into changes.
But we do know some sane things:
- The stakes couldn't be higher.
- Good: Don't allow inbound SSH connections, even through a fancy $100k firewall.
- Best: Don't let people login with SSH (treat SSH like we treat the serial port: a debugging option of last resort)
SSH is kind of a swiss army knife. But 1000x sharper ;) The delta I'm speaking of would be to have bespoke tooling for different needs. And the tooling for each purpose could have appropriate, structured logging and access controls.
With SSH you can do almost anything. But you can imagine a better tool might exist for specific high-value activities.
Case study:
Today: engineering team monitors production errors that might contain sensitive user data with SSH access and running `tail -f /var/log/apache...`.
Better: Think of how your favorite cloud provider has a dedicated log viewing tool with filtering, paging, access control, and even logs of what you looked at all built in. No SSH. Better UX. And even better security, since you know who looked at what.
---
There are times when terminal access is needed though. SSH kinda fits that use case, but lacks a lot. Including: true audit logging, permissioned access to servers, maybe even restricting some users to a rigid set of pre-ordained commands they are allowed to run. In that cases, a better built tool can allow you to still run commands, but have a great audit log, not require direct network access (mediated access) to servers or internal networks directly, flexible ACLs, and so on.
You can lock down ssh remote commands, and of course require proxied connections. You can include full auditing of command execution.
But that's either the same with any tool regardless of whether it's commercially supported / FOSS / made by anonymous devs or not. If anything, FOSS is easier to audit.
Historically, it's been pretty good though.
If you would consider a commercial alternative, consider how much you would need to pay to get an actionable warranty, and consider if you could find someone to do a warrantied audit of OpenSSH (or especially of OpenSSH in your environment) for that amount. It might be a better use of your money.
Downvoters, what are you trying to achieve?