xz has shown how a dependence on unpaid volunteers can cause major problems | Hacker News Reader

xz has shown how a dependence on unpaid volunteers can cause major problems

twitter.com

55 pointstosh31 comments

Actually its shown how a dependence on people whose identities have not been confirmed and who can merge code without review can cause major problems. If you restrict it to people who get paid then the nation state or other bad actor will simply send someone to get a job that gives them the privilege or blackmail someone. who already has the privilege.

Indeed.

I'm very frustrated by all of the "this has shown how fragile FOSS is" hot takes I see appearing. I think it has shown us how awesome FOSS is: If this were all closed source software, chances are we'd never have known about the issue, nor would an entire community now be able to have an open conversation about how to mitigate it in the future, applying their combined brain power to the problem.

And there's a fair amount of obvious steps to take now:

- Re-visit the pro/con of anonymous contributions. Re-visit identity verification. Maybe do a SPDX-like thing for governance/contribution work flow mechanisms for project to report how they work and what stance they take, to allow downstreams to risk-rate. Some projects will very much want to continue to allow anonymous contribs for excellent reasons, and that's fine.

- Phase out build and module systems that drive published tarballs differing from repo tags.

- Have distributions verify that tarball contents match repo tags on import of new versions. Yes, this requires that projects have a repository that is reachable online. I think for critical SW that's OK.

- Don't accept build systems that silently disable security features when dependencies are unavailable. Not building a security feature should always be a decision point and manual opt-out.

- Come up with better metrics for maintainer burnout and heartbeat signals.

Who watches the watchers?

Moreover, identities on the internet are fairly easy for motivated individuals (or moneyed organizations) to fake.

Finally, who's going to build and maintain everything you've proposed?

Companies need to sponsor work done on FOSS. FOSS maintainers need to reject requests that don't benefit the broadest possible audience unless they're receiving a sponsorship to do it. And that's just at a bare minimum. Without aligned incentives, I personally think any attempts at fixing things will be ineffective.

You're somewhat right, somewhat wrong and somewhat as out of track as Ffmpeg: Lasse Collins should have been a team, a team that meets, so the problem of "unknown rude complainer" pressuring for "some guy working without scrutiny on your behalf" would be mitigated. But nothing impedes that your well-known lads can also work for the NSA or even for Microsoft.

I'm adamant on thinking that first and foremost there is a smorgasbord of crazy designs at play: a build system so messy you can slide in binary injections, an init system with dependencies, an execution chain where you can easily rewrite whatever else's functions. It's fine if somewhere down the drain someone churns garbage, the priority has to be that it should be easy to detect it, I find that it's dumb and sub-par to rely on a fragmented and laggy circle of reviews, pre-releases and releases, (yeah, everything handled by thankless and lone volunteers makes it worse), for protecting our systems.

Even if all contributors have verified identities, sign their commits, etc it isn't enough. Consider the real-world case where a developer's laptop is hacked and the attacker subtly modifies their local source tree to introduce a vulnerability that is unwittingly signed and committed by that developer. I've seen cases like this in the wild caught in code review.

It always made me wonder how many such cases don't get caught, since code review isn't consistently thorough.

Identities are even easier to fake for nation state.

I fail to see the "problem with being an unpaid volunteer" angle in this story, I wonder why ffmpeg staff worded it that way. Open source always had this "danger", yet the massive benefits made corps never question their use day by day. And with "massive benefits" I actually mean "it holds everything else on its shoulders".

It explains right in the tweet. Big companies take unpaid work and start to rely on it without giving anything back, then demand fixes as high priority.

Also, being unpaid implies that one has to do other things to get money, so that leaves limited time and energy to put towards the free project. So when pressure starts coming in from outside, the maintainer(s) don’t have time for real reviews and accept code without fully vetting it.

If maintaining this code was part of a paid day job, there would be more time, energy, and incentive to review it more completely.

Open source code is generally supplied as-is without any liability acceptance or warranty. If your business relies on it and it has bugs then that is the business's problem to deal with.

But there were paid "volunteers" who didn't do their job either (/cough lgtm). If anything, this incident shows a large amount of complacency with pull request reviews and contributions in general.

b-b-but if I actually do a real code review then the business side won't get what they want by sprint's end!

The problem is not being unpaid. The problem is lack of verification and control, which itself result from lack of help and participation. You won't fix these problems by throwing money at it.

And will people do verification and control without pay?

Let’s say that that’s the problem. Who are qualified to solve it?

American programming at least is centered around a few hubs where the programmers make great wages. They work for companies which make a lot, lot of money. Much of the complaining is about professional programmers who do OSS as a hobby.

So none of the parties are lacking in resources. And no one is being coerced.

I bring this up because some problems are about exploitation and coercion. Then the soft (unreliable) solution is about morality (don’t exploit), while the hard (reliable) solution is about laws and law enforcement.

But Ethics doesn’t seem to be the appropriate domain. And not Philosophy in general.

I’ve wondered for years now: where are the economists? This is a problem of assigning resources. It’s a problem of incentives. It’s a problem of dependable and proper bus-factored maintenance. Isn’t this a kind of problem that they are trained to solve? Concretely: you have an unreliable (volunteer, small) resource and large entities that rely on it. How do you bolster that resource?

And I don’t mean in the vulgar sense of “give X and Y more money”. I mean: the current status quo is apparently unreliable, so how should things be organized in order to promote more reliability? That does not have to involve money.

This isn’t some academic exercise. Real money is apparently on the line, indirectly. The stakeholders are powerful. Why can’t the relevant professionals solve this?

Don’t get me wrong. I’m the kind of annoying person who believes that a lot of economics is mostly about propaganda. But how is this not a problem that economists can solve?

Well, with free products and services, our policy is "if you're not happy, we'll give you double your money back"

Exactly this. Corporations can pay up if they want these pieces of critical infrastructure to be more robust. Fully leaning on the passion of people in our profession to do unpaid work because we like the craft is... uncool.

In this case at least reading through the timeline it sounds like the bulk of the discovery of the vulnerability came through paid folks though.

"shoulda put a ring on it"

Microsoft aren’t paying, but are still demanding responses to their problems. Now sure you can ignore it (and the ffmpeg lot are a hardened bunch), but I can see how a lone developer can feel pressured by leviathans

Microsoft are being the twats here, not the ffmpeg devs

> The xz fiasco has shown how a dependence on unpaid volunteers can cause major problems.

To be honest it has also shown that it wasn't all too wise at all to introduce a completely needless dependency on a rube-goldberg piece of software in what was otherwise something created by security-minded people.

    openbsd -> openssh

vs:

    systemd Linux distros -> openssh -> (lib)systemd -> endless potential venues for backdoors (oh, look, we just found one)

I'd give Theo de Raadt a thumb up and Poettering a thumb down here, again.

Seriousy, why the fuck did Linux have to be polluted with that gigantic squid that systemd is, throwing is tentacles about just everywhere. I know, I know, there are distros out there not using systemd but they're rare.

I find it very weird that so many people keep downplaying the role of libsystemd here (makes me wonder what their agenda is).

> I'd give Theo de Raadt a thumb up and Poettering a thumb down here, again.

Poettering didn't talk distros into patching openssh to call libsystemd's sd_notify(), and if you look at the PR for dropping libsystemd's linking of liblzma you can also see him take a stance against the dependency there.

It's not about volunteers, it's about who is doing the work. It's another symptom of the movement to make the Internet a "zero-trust" environment over the last 20 years.

For some reason, people want to only interact with text on a screen. Who's taking this crypto? Who cares? Who's managing this website or forum? Who cares? Who's maintaining this software package? Who cares?

We're semi-hairless apes with brains adapted to living in social groups of 150-300 people, and now we have placed so many of the daily transactions of life into this system that could theoretically make the social group the size of humanity. We've fooled ourselves into thinking that if we just make anonymity a feature of the system, bad actors can't hurt us.

If Microsoft wants to use ffmpeg, then they should have to have people on staff who can fix it when it breaks. The whole point of FOSS was that you could have the source code to things you rely on. If there is a problem, you can fix it or pay someone else to fix it.

I mean, I'd say that this fiasco showcased the significant benefits of passionate volunteers doing great work in the open. And yeah, someone at MS may have made a faux pas on the ffmpeg forum but also someone at MS was the one who discovered and responsibly reported the xz vulnerability. Corporations aren't monolithic.

If this had happened and someone's livelihood or next promotion was on the line, the amount of forensics and publicity would likely have been so much less open than it has been in a product driven by community and done in the open.

The xz incident was handled far better, faster, and more open than the Solarwinds incident. If anything, this proves that more eyes do help.

This is actually a bit ridiculous - having helped clients more than once, this can be the same for employees, contractors and partners. Nothing new here.

This is business negotiation over the price and terms. Apparently they offered a long-term contract to Microsoft and got a lower counter-offer for a short-term engagement. If that offer is too low, maybe see if there are other terms that both sides would accept?

Pelying on unpaid volunteers, particularly when trillion-dollar corporations expect urgent support for free. It's unsustainable and underscores the need for a more reliable system that doesn't exploit volunteer labor.

Relevant.Ticket[1]

[1] - https://trac.ffmpeg.org/ticket/10341

With Elon Musk chiming in with some suggestions

Even paid developers are vulnerable to a nation state bribing or blackmailing them into accomplishing their objectives.

I'm truly staggered by the amount of commits here blaming the issue on the "lack of identification" or "dependency on unconfirmed identities". Even worse, some posts even proposed "security audits from trusted companies".

I knew people would be reactionary, but not "unwind most of the free software movement ideals" reactionary.

  1. After $EVENT, we are in a horrible state of affairs. Something must be done.
  2. $THING is something.
  3. Therefore, we must do $THING.

(We don't examine whether $THING could've prevented or alleviated $EVENT - there is no time.)

Reply on news.ycombinator.com