Only a few vendor-paid developers do almost all open-source work | Hacker News Reader

Only a few vendor-paid developers do almost all open-source work

infoworld.com

151 pointstomxor43 comments

This is hilarious:

>> It is fair to say that for almost all of the projects in the CNCF, specific vendors account for most of the development work being done.

> Not just “many” open source projects—all of them.

The author moves from an analysis written about CNCF projects to generalize a conclusion about all open source software. What a misleading title and content, it remains hanged without any evidence on those claims.

The original "study" http://redmonk.com/fryan/2018/04/03/who-contributes-an-analy... has an appropriate title:

> TL; DR: The vast of code contributions to each CNCF projects outside of Kubernetes come from a small subset of users and companies.

InfoWorld sensationalized the title.

Matt is generalizing outside of the CNCF though. He specifically mentions Linux.

Personally, I wouldn't have used the term "all" absent having actually looked at the numbers myself. That said, it's probably at least mostly true, especially if you look at the subproject level in the case of projects like Linux and OpenStack.

> it remains hanged without any evidence on those claims.

While I get where you are coming from, give me a counter example/evidence.

My observation of the open source world is pretty much in sync with the generalization of this article. Even if you might find some good counter example it most likely will be the exception to the rule.

I'm having trouble understanding this view. Of course most of the big projects like Linux or the browsers have corporate backing, but that ignores the huge amount of work done on much smaller projects by unpaid devs. I don't understand how anyone could say "I can’t think of a significant counterexample" as the author does; surely that just reflects his ignorance of anything beyond big corporate projects. Most of the programs I'm interacting with daily are built by volunteers; that's pretty typical of the Linux ecosystem.

Sure, Firefox has corporate backing, but the half dozen or so extensions that make the browser useful for me are all built by volunteers. uBlock, uMatrix, TreeStyleTab, Stylus, etc etc.

My backup software, borg? Built by volunteers. My music software, mpd? Built by volunteers. GIMP, useful for quick edits? Volunteers. My text editor, most of the games I play, terminal utilities, torrent clients? Volunteers too. Even the enormous amount of packaging work and bug triage done for my distribution (Arch Linux) is done by volunteers.

Even KDE and Gnome, I suspect, don't have many devs hired by corps to work on those projects. (Though I might be mistaken.)

Maybe by lines of open source code, rather than number of projects, you'd have an argument that corps do most of the work. But when it comes to the program ecosystem on Linux desktops, you're completely surrounded by a bunch of volunteer projects.

KDE has always had paid developers behind it, but only a handful.

Different companies used to employ people to work on KDE including TrollTech, Nokia, Novell, Blue Systems, Canonical, Red Hat.

I think these days it's only Blue Systems that does so. Maybe also Novell but I could be wrong.

The overwhelming majority of the work is done by community and volunteers.

Gnome always had more corporate backing, primarily Red Hat.

Definitive statements are easy to pick apart, but I think he's basically correct. Pretty much all the open source we use are made by a tiny handful of people. It makes sense because most software works this way anyhow, and when there are large groups involved it takes intense coordination.

The more existential question is: are those 'peripheral' additions critical? Because maybe that 'last mile' is superfluos, i.e. minor bugs and 'nice to have' features, but it's possible they are key contributions.

It's possible that the 'key contributors' are like 'managers' or 'curators' of the project, bringing in the work of others, making decisions about it, possibly borrowing ideas from the community and 'implementing it themselves'.

Yes. And a lot of projects these days emphasize "casual contributors" who work on small external modules, documentation, management (which, as you suggest, isn't explicitly factored into the stats), etc. As the piece suggests, it may be wrong to view open source through a lens of free, happy, and democratic community contributions but it would be equally wrong to say that it's just like proprietary software development.

> It makes sense because most software works this way anyhow

What does that mean?

It means most software projects are actually written by small teams of under 10. When you don’t count all the project staff, testing, operations, user support, etc.

Softwareis typically just lots of small components linked together. Especially super generalizable stuff like popular open source.

Open source contributions are actually mostly not from the project in question. You'll almost always depend on a mountain of other code, in the form of the kernel, software libraries, user utilities, and so on. A lot of these are specialized projects and have their own core developers. Your project may be possible without them, on another platform for example, but in Linux there's still thousands of random devs contributing to a whole.

Cloud software has suffered from NIH for a good long time now, and required proprietary APIs to function. You have to "integrate" your app with another app, it never just has implicit compatibility. Sad really.

That's a good point although it's not wholly unique to open source.

In addition, container platforms for example may be made up of a fair number of different projects including container orchestration, the base container capabilities, monitoring, logging, CI/CD, etc.

That claim that "vendor-paid developers do almost all work" also fits the Linux kernel. The kernel is very large and has many contributors but work done by random volunteers is dwarfed by that done by large companies. This is visible in contributor statistics: https://lwn.net/Articles/750054/

For the kernel the "community" is to a very large extent a community of companies.

I get the impression that many of the kernel developers in these companies would be active in the community no matter where they worked. For example see this interview with one of the names on the list:

‘I can't imagine doing anything else for a living after having looked into most kernel subsystems at some point over the last 15 years.’

https://www.linux.com/news/30-linux-kernel-developers-30-wee...

The article uses "commercial" when it should really use "proprietary". The open source projects mentioned are all very much commercial software.

I'd rephrase that slightly. In many cases, there is commercial software that makes use of many of those open source projects.

Infoworld: "We care about your privacy" That is the headline, in an interface which allows me to scroll 32 times so that I can disable sharing my personal information to all the ~300+ different parties they share it by default.

>>The irony is that what makes open source work—and differ from commercial software—is that only a few developers do the major work on any project.

Comparing Open Source vs. Commercial software - that's the fail of the article right at the beginning... Apple vs Oranges... What has one to do with the other? Maybe they meant proprietary?

I'm a bit surprised that Matt used that terminology. It's pretty obvious from the context that he's using commercial as a synonym for proprietary and he certainly knows that's sloppy IMO.

Yes, good point. I've seen a few well-known/famous tech writers make that mistake, either unknowingly or carelessly.

Open source or free software (as in speech) vs. closed source (proprietary)

and

commercial or paid / non-commercial or free software (as in beer)

are 2 X 2 (orthogonal) variations, giving 4 possibilities in total.

Odd that not many people see this. Maybe sloppy as ghaff said.

Correct me if I got it wrong, anyone, that's my understanding of "free as in speech" vs. "free as in beer".

I prefer libre (as in speech) and gratis (as in no cost) but it might be because I'm Spanish.

The thing is, they are conceptually orthogonal, but in practice there's strong correlation. It might be for many reasons, but I'd guesstimate 90+% of open source is also non-commercial.

Are you implying Linux is not commercial? What about ElasticSearch? GitLab? PostgreSQL?

I don't know about those legal affairs, but they are still only a very, very small part of the open source (in amount of repositories). For instance, the whole npm is open source (ISC) and non-commercial by default, which is currently the largest package manager.

>but in practice there's strong correlation. It might be for many reasons, but I'd guesstimate 90+% of open source is also non-commercial.

Your guesstimate may be right or wrong (also see sibling heavenlyblue comment), but my point was more that those writers I mentioned, do not seem to realize that open source or free software can be sold as well. They seem to confuse FOSS as being *synonymous" with "free as in beer".

And even if "90+% of open source is also non-commercial", that can be just because many people do not charge for their open source software. There's no rule that they cannot, though.

I agree, I'd be the first one who would like to get some money for my open source :)

I am just saying that if there's such a strong correlation it's normal for some people to mix them up.

Really good insights on how this is tots fine and works well, but I think it also misses something key:

As company, you can make money by giving something away for free.

It may also be a tool to to push out competition. Keep your own money making stuff proprietary but make everything else open source so others can't survive. this seems to be the Google and Facebook strategy.

If you make the stuff that powers your way of doing business free, you'll push out everyone who tries anything different. If your way is rule, you're king.

Under this analysis, some portion of the Open Source efforts out there fall into a very old tradition that includes promotional giveaways, loss leaders, and so on. It is the application of old principles to a new arena.

That isn’t key. Making money by giving away something is free is physically impossible.

Therefore there is something critical you are missing about what is going on.

It’s only key in the other sense that it will unlock a door to greater understanding if you seek out what underlies the contradiction.

Well, it kinda takes more people working part time to do the same amount as someone working on it full time. Throw in the fact they also probably get paid. Yes, the will code more the most volunteers. Sadly, volunteers who would contribute more have to eat too.

Although, I find the statement "all" is a bit misleading. I know of quite a few projects done by people on the side.

While each individual project may have a core group of committers, are they the same people each time? This conclusion was not clear to me from the article, though the Hacker News title implies it.

If that is the case, it goes against my experience with open-source. True, individual projects have core committers who drive the project forward - but the ecosystem that springs up around a project doesn't typically derive entirely from this core subset. NPM is likely the best example with 350,000 packages existing on the platform alone with the distribution of developers to packages being fairly uniform (reference: https://github.com/substack/npmtop/blob/master/README.markdo... - mind that the numbers are from 2011, but even back then it was highly uniform).

As others have noted, the article is based on a study that only looked at a small subject of OSS projects, so it's not very solid. But the results fit with many commenters' perceptions. It's surprising that so little is known about the dynamics of how OSS gets written, given how crucially important it is to the global economy these days.

(Even) more broadly; I wonder if a similar system will wind up dominating in some fields of science. There's been a bunch of cases where prominent research papers in AI or biomedicine have been published by entirely for-profit companies; one supposes they benefit in the same fashion that more traditional software companies benefit from publishing open source work, namely by building a reputation and cultivating a larger ecosystem around their expertise.

Not much is known about the reality of open source because people do not want to know. This is the biggest story of the last decade: a few major technology corporations have banded together to give away billions of dollars of software completely for free. It is, I suspect, the greatest act of charity our planet has ever known. It's not difficult to see how virtually all of the major trends from the rise of China to the ever expanding proliferation of smartphones and IoT are being driven by this.

The disconnect between the narrative around technology companies and and the reality is due to a severe cultural blindspot. It would simply never occur to most people that private companies would compete to give way most of their products. Even people who should know better don't want to admit that much of the software they consume (for absolutely zero dollars!) is provided by the same companies they love to whine about. We seem to be missing a whole new phase of capitalism happening right under our noses and there's really not enough research into it.

you know what would be interesting... is if there was some kind of equivalent for dtrace or strace or gdb.. but instead of collecting statistics on performance or call trees of functions, it could instead tell you statistics on who wrote the code that your machine is running, line by line.

Could modify debug symbols to contain git blame output then collect it in a profiler.

We'd probably just learn who wrote the busy/wait kernel loop.

This is by project instead of by machine, but CNCF develops DevStats to show a number of statistics on each of our projects, including the companies behind the development.

https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1

Disclosure: I am executive director of CNCF.

Keep in mind that many of these corporate employed developers would likely contribute to the same OSS projects as a part of their job regardless of what company they happen to be employed at.

I don't think this is true.

First of all some workplaces won't allow it.

Some companies don't even run on OSS.

Some companies will work on it but not contribute back.

Some companies won't have a budget to fund random / semi-related OSS project development.

There are examples. For instance, both Linux (Linus Torvalds) and Python (Guido Rossum) have been steered by original contributors who changed employers several times, always still retaining their role as primary developer/coordinator for that open source project. Not saying it's the majority, but Linux and Python are two pretty important examples.

Those aren't relevant examples for drawing a conclusion about the relationship between the average contributor to Linux or python and their employer.. Or for the lead of an average project and their employer.

You made this point better than I could. If anything GP mentioning _important projects_ is by definition the exception.

There are doubtless some prominent developers on a number of major open source projects who could find employment at a variety of companies without changing much about what they work on.

They're a minority.

Most companies expect developers to focus on what's important to their specific business and not what was important to their previous employer's business. They can't just work on whatever they feel like, especially on company time.

Reply on news.ycombinator.com