Hacker News Reader

Comment by michelsedgh | Hacker News Reader

I understand what you are saying and for open source community to have a good reputation and keep its standards high, these important packages have to start being actually maintained by the enterprises that depend on these. Most of businesses use Linux for their servers mainly Ubuntu/Debian and its just crazy for them to be vulnerable in such ways. Imagine how many other places backdoors can be installed even right now, and many of them won’t be visible and wont have performance issues like this one. 0.5s is a big deal in software and if it was 50ms or 5ms no one would have known this possibly for years. I wonder how safe the software we use actually is.

View on HN

These open source projects have a good reputation and high standards because the developers freely volunteering their time care about the project. Forcing these developers to give their projects to Big Corp may not be the big improvement you are imagining.

Software may have backdoors, I'm not convinced this couldn't have happened if a large company was managing the product. AFAICT, open source software has a better security track record, in general.

IMHO, if it is anyone's job to look for and prevent these kinds of backdoors, it's probably companies like RedHat and Canonical. They bundle the tools with their products and they charge money for support.

Backdoor detection is an unsolvable problem both in theory and in practice. We are talking about carefully hidden obfuscated backdoors in MLOC codebases, which will have unexplained binaries and arcane build scripts. No company, not even Google, can guarantee that they'll be able to catch every backdoor hidden in some arbitrary codebase.

The only practical solution is for these companies to rewrite the core packages from scratch, written from the ground up so that they are easy to audit - no unexplained binaries, no arcane build scripts, etc.

> AFAICT, open source software has a better security track record, in general.

iirc Studies was done on this and there was no measurable difference in security issues when it comes to open vs closed source.

Just because the code is visible doesn't mean much, as you need to have the right eyes to actually notice security issues and a lot of open source projects don't have this.

Also closed source projects have the advantage where if they have a massive company behind them, that company has the resources (i.e. $$$) to hire highly specialised people to look the security of the software. Open source projects usually don't have such resources, heartbleed and xz are indicative of that.

If you want professional developers pay professional rates. A doctor asks for a hundred bucks for 15 min.

Yes and people are paying top money to Canonical and RedHat to maintain these software don’t you think? They have huge enterprise contracts and I feel like this should be on them? Im not blaming anyone it just goes to show how vulnerable they are. Also other linux distros, Fedora, arch, etc. they have the money and should make their software safe don’t you think?

Fedora is supported by Red Hat. And yes, I do think there is something to the fact that Red Hat is selling businesses commercial support for projects they have no expertise in, which probably means some of the pressure flows downwards from Red Hat's clients to the unpaid maintainers.

Arch, (Gentoo, Nix, Mint, etc.) have no commercial aspect on the other hand and certainly doesn't have the money, but those distros are offered on the same "as is" basis as the downstream software.

Yes and haven't Canonical and RedHat and others already reacted and their users are fine?

I see no problem.

First, the problem is the reputation hit and the trust going away. Right now on Hacker News 2 maybe 3 of the posts on front page have been about this since the news broke out. Also when trust goes away, its very hard to come back. Secondly, the main problem that I see is how many other backdoors/dependencies are vulnerable that we might not know? They might not have performance issues. Also if this went unnoticed, in the long term it might have found its way and actually compromised people. I’m glad this was taken care of and no one was compromised.

Is there some implied proposal I missed?

https://news.ycombinator.com/item?id=39904034

OpenSSH is developed by the OpenBSD project and I have a lot more confidence in them than in any random "enterprise".

The issue in this case is that Linux distros took it upon themselves to alter its code base by linking in libsystemd (thus also liblzma) for the dubious benefit of better systemd integration, which comes with a generous helping of attack surface.

> Yes and people are paying top money to Canonical and RedHat to maintain these software don’t you think? They have huge enterprise contracts

Money alone makes poor filler material to patch such holes. Microsoft famously doesn't make their stuff for free, and yet somehow Windows systems have gained this reputation of having holes such that a train can go through.

People build trust, take shortcuts, cut corners all the time. It's how we are wired. Doing otherwise is an energy sink for our bodies. We don't make ourselves overspend energy when we can.

I'd hardly call a $400 per year per seat contract top dollar.

There is software out there that is both much worse and costs in the tens to hundreds of thousands per seat per year.

There are economies of scale for widely-used software.

Reality disagrees.

Software isn't tooth brushes. The simpler and more specialized it is the cheaper it is to maintain. The more complex it is the harder it is to maintain.

Software R&D is a fixed cost. An ERP package with its at most 10,000 installations amortizes that cost on far fewer users than Linux with its millions of licensed installs.

> Also other linux distros, Fedora, arch, etc. they have the money and should make their software safe don’t you think?

Nope. With open source the user is responsible. It's been this way from day one. Come to think of it, almost every piece of software I've ever used that is commercial has a no warranties and not suited for purpose clause in the EULA... because all the components the vendor used have that clause in the license to the vendor.

There's no connection between how much money somebody charges for their services and the quality of the services that they in turn provide.

Much of the worst professional service I've ever received, despite paying them a lot of money, has been from the example you gave, doctors.

Doctors aren't alone, of course. The most expensive restaurants I've ever been to have been some of the worst dining experiences I've ever had, while friends or relatives inviting me over for a backyard barbecue have been among the best, for example.

I've also found this to be the case for lawyers, accountants, teachers/professors, mechanics, and other well-compensated "professionals".

Software hasn't been an exception, either. I've had more success with free software than I have had with the equivalent paid offerings many times over.

In general, I've found that service quality is far more tied to the provider's level of passion for the task at hand, rather than anything to do with the amount of financial compensation that may be involved.

The software we use is not safe at all. If we don't have an idea what goes on in an open source projects, how could we even begin to know what goes on in a proprietary binary, or a service? People's dealings operate on trust, that's how, and trust can and will be abused, simple as that.

>Most of businesses use Linux for their servers mainly Ubuntu/Debian

Businesses should calculate with the risks, and also, use Long Term Support versions of software, not unstable bleeding edge fresh from the oven ones. Which they do. And so they were not vulnerable at all.

It's not the maintenance of the projects the enterprises have to fund (though that is certainly welcome if the mantainers are willing) but the slow, painful work of vetting the code.

Google does that with Project Zero but few companies are wealthy enough to afford that. The way out is economic, not technical: insurance, and mutualizing the cost of security audits. I wrote up my ideas on the subject here:

https://blog.majid.info/supply-chain-vetting/

What are you actually proposing here? Any Ubuntu/Debian distribution contains hundreds or thousands of packages and transitive dependencies - it's unreasonable to expect corporations to contribute, even financially, to all of these.

This is supposed to be what services such as Tidelift provide - companies pay them, and OSS maintainers sign up with them for funding (and other support? unsure). But it's tough to apply this through the whole OSS ecosystem. To me, this should be what Canonical's support contracts should be for. They're perfectly positioned to help mediate these types of issues.

Im proposing that high security packages like this that effect ssh and main parts of the OS critical for security and safety should be maintained by actual enterprises not by one developer. Im not saying they should maintain everything, only the high importance parts of it. They should make sure that one bad actor cant compromise the whole ecosystem. I really don’t think what im saying is that far out of reach or that crazy. Edit: also, why should a package like ssh have so many random dependencies? They should make sure these kind of packages have the least of dependencies especially on these kind of packages that have one maintainer to reduce the security risks?

why should a package like ssh have so many random dependencies

that's the critical question here. it's not that enterprises need to take over maintenance of xz, but that the critical packages and their dependencies need to be audited on a regular basis. the development of xz is ok. what it needs is help with code reviews. and if it can't receive those, then it needs to be removed as a dependency of openssh.

Yes, exactly, these high security packages that every distro uses must be very tightly maintained and looked over with the least dependencies. SSH is one of the most important packages in security and it just feels clumsy that they are using such packages like XZ. Just WHY? Who made that decision? Why doesn’t all the distros work together maybe to maintain the security? Why depend on one person that does this as a hobby?

> Edit: also, why should a package like ssh have so many random dependencies? They should make sure these kind of packages have the least of dependencies especially on these kind of packages that have one maintainer to reduce the security risks?

Using standard libraries for common stuff like compression, cryptography and whatnot is vastly more preferable over everyone shipping their own crypto, or worse, patches of crypto (see the Debian SSH key vulnerability of 2008 for an example [1]). For protocols it's in the end just as bad, it's a nightmare to keep different versions of the same program to be able to talk to each other, but now imagine a literal ton of programs who all have a wild mixture of statically shipped libraries, homegrown stuff that has barely been tested... no, just no. Not a world I'd like to live in.

[1] https://jblevins.org/log/ssh-vulnkey

why should a package like ssh have so many random dependencies

It's worth noting that vanilla upstream OpenSSH DOESN'T depend on xz. The dependency was patched in by the major distros to better integrate with systemd - and it's (AIUI) a transitive dependency from systemd - even the patch doesn't use it, but it gets pulled in with other systemd stuff.

That is worth noting, thanks for posting this and letting me know.

vanilla upstream OpenSSH DOESN'T depend on xz

that's what i believed as well, but i didn't take time to verify. and i didn't know about the details. thanks. as you describe it ssh still doesn't depend on xz (and why would it?) so part of the problem here is software architecture.

how is it possible that a seemingly unrelated dependency somewhere within systemd can affect and be exploited through ssh directly?

shouldn't it be possible to keep that separate?

doesn't openssh itself already implement some form of privilege separation?

how does software architecture here and in general need to change to prevent things like this?

i am sure somewhere these questions are already being discussed. i'd appreciate any pointers.

Actual enterprises do not have a better track record. I would go as far to say that if Big Corp maintained an SSH client and server then a lot of distributions would shun it out of distrust. How many products from actual enterprises phone home with telemetry or leak data in other ways?

> high security packages like this that effect ssh and main parts of the OS critical for security and safety should be maintained by actual enterprises not by one developer.

I don’t remember the company names, but I guess people will remember the incidents.

- Solarwinds have been breached end to end.

- A company has been bribed (forced?) to ship backdoored encryption algorithms.

- A network hardware supplier’s firmware had been backdoored by Chinese IIRC.

- NSA backdoored national standards.

- Microsoft has been breached end to end.

In short, even if you’re a company, you’re one NSL, one bad actor, one misstep away from “total pwnage”.

I trust some individuals for developing critical software than entire “enterprise”s.

Everything is as strong as their weakest link.

> What are you actually proposing here? Any Ubuntu/Debian distribution contains hundreds or thousands of packages and transitive dependencies - it's unreasonable to expect corporations to contribute, even financially, to all of these.

Why is it unreasonable?

I actually see it as very reasonable. You use a package in a commercial distribution, you use aggregators like Github sponsors to pay the maintainer (a subscription, not a one-off payment!). What's unreasonable about that?

See the rest of my comment, where I propose using an aggregator. I think it's unreasonable to expect everyone to engage in support contracts individually with every maintainer.

I don't really see how paying the maintainers helps against insider threats. The users still has to ensure that the software is safe.

Paying the maintainers is nice, but I think the megacorps and governments has to fund an organization that does security audits of all the sensitive packages, instead of heaping yet another job on the shoulders of the maintainers. It's especially important to not saddle the developers of open source with red tape, most of them would think that getting paid is poor compensation for the lost freedom.

Proprietary Unix vendors used to develop all the system libraries and utilities. It's very much doable, especially if the responsibility is split across Linux vendors. Vendors need to take a critical look at what they're using, who develops it etc.

Open source community currently discovering that charity doesn't work and why governments need taxation to provide services. Libertarians in shambles.

Well, I'm certain Facebook would be glad if xz is replaced everywhere by zstd which they maintain. You can't force them to pay for xz when they already maintain a replacement.

Reply on news.ycombinator.com