The strange economics of open-source software
philipotoole.com
philipotoole.com
Most people, when understanding economics tend to make the jump directly to money/cash when explaining prices and market action.
But to be correct you have to look at utility. There's little question as to why companies install open source software - the TCO can be lower. But why do people contribute their time in building it?
The answer is that the reward of being part of an OSS team brings great utility to a developer, and this is true even for marginally successful OSS projects. So people exchange their time for intangibles like reputation, credibility markers ( the author casually drops in which projects he is involved in) as well as the ability to leverage involvement into actual cash like well paid consultant gigs, books or successful blogs.
So there isn't any strangeness going on at all. OSS makes sense from a company point of view, and it makes sense from an individual developer point of view.
Why is open source good for everyone? You didn't explain it, you just cast it in certain terms, which is useful for distilling the problem, but certainly not for solving it. You gave your guess why it improves the contributors' utility, but you haven't really proved it. It's just a guess. I can guess, too.
My guess is that open source software reduces the transaction costs of acquiring and fixing software for corporations, but I could not explain to you with neoclassical economics why developers work on it without being compensated. It should be a tragedy of the commons, but it isn't. Open source software is developed at the fraction of the cost of closed source software and either competes or completely dominates it.
I don't believe in Liberal organizations of economy though, so it's not a big deal for me.
My recent open source work is partly a response to a frustratingly anti-refactoring employer. The money was worth it, but I needed to recharge my "you can actually make progress as a good pace on a project that is well maintained" batteries after work.
I don't at all agree that classical economics as a specific school is particularly focused on utility compared to others. Wouldn't that be a little like saying high school algebra is closer to set theory than say abstract algebra?
To establish a correspondence between utility and the real world, I think it is worthwhile to acknowledge that an understanding of what predated currency has shifted away from barter economies and toward economies that traded influence and status. Or, similarly, take a look at this story from a few weeks ago[1], where the consistent theme to me seems that people's behavior occurs within a context, and that anthropology might be as good a way to explain things as a highly simplified reward and punishment framework. That is, utility is maximized by acting consistent with your perceived identity as much as through projections of rewards from various choices.
So, while I don't think the concept of utility by itself says that status and identity play the primary role, I also don't think it is implausible.
And yet, you're right that open source isn't just hobbyists, and I doubt contributions from corporations contribute to prestige in any way that is valuable, so as you said:
>My guess is that open source software reduces the transaction costs of acquiring and fixing software for corporations [...] It should be a tragedy of the commons, but it isn't.
— leaving us with a poorly understood phenomenon. Is this just a moment in time specific to the age of a current generation of programmers, that won't last, or is there a dynamic that is more sustainable?
Why is this? Your average software company runs their R&D department on about 10% of their earnings. Yes, I pulled that number out of my nether regions, but in my experience that's about right. If you do a google search for R&D as a percentage of earnings, you will see lists for large companies that range anywhere from 2%-30%, but they tend to be on the lower side in my experience.
This means that for a software company to produce, package, market and sell a software product, they have to charge about 9x what it cost to produce it. If we factor in profit, then we're pretty close to an order of magnitude difference.
With open source software virtually all of that overhead disappears. If you are writing software that you already needed anyway, then the cost is very low -- even if nobody contributes to the project from the outside.
With a traditional software sales model, you divide the cost over many customers. So it might cost you $10 million to create and sell, but if you have 100,000 customers, you can charge $100 each and break even. With an open source model, each actor builds pieces that they want and care about and uses other pieces for free.
The advantage to working with the open source model is that the overall cost is an order of magnitude lower because you cut out all the sales, marketing, support and associated management. Not only that, but the pieces you pay for, you get to dictate exactly what they look like (within reason ;-) ). So essentially, you are getting your software at a tenth the price and the bits you care about, you are getting custom made.
Of course, you usually have to spend time packaging and documenting, but this cost is quite low. The reputation you get for doing it is often more than worth it (especially if you look at the cost of hiring recruiters). It would be interesting to see a study to see if sponsoring popular open source projects leads to lower attrition for companies. I suspect this is true, but have no data.
Of course, this sucks if you are in the business of selling software. If you are in the business of selling services enabled by software, then open source is absolutely king.
Individuals still contribute quite a bit to open source projects and have their own internal drivers for doing so. More and more I've seen advice that in order to show your worth to an employer it is best to work on open source projects and to have a portfolio (I admit to giving this advice myself, from time to time). Lately, though, I have found that the majority of important open source projects are funded through commercial entities. The last time I looked, the Linux kernel was about 75% funded through commercial entities. On the web side of things, it's actually getting difficult to find popular projects that aren't mostly funded by one or more organizations. I think the era of running large software projects as a hobby is coming to an end.
The point of utility is that everyone is different. They will all behave in chaotic and unpredictable ways. The only way to make sense of why people exchange is to make the assumption that each participant in the exchange gains something greater than what they give up. Since the thing they gain and the thing they give up are always going to be different, it has to be distilled down to something. That's utility.
You say that I haven't solved 'the problem'. I say there is no problem to be solved. All we can do is observe the actions underway because there is nothing to fix.
We can't say why the contributors utility is increased, because by very definition, each persons utility is different. You can only infer or measure the subset of those, and make observations. Utility is the abstract class of individual preference implementations. There is no more point in trying to determine the cause of each individuals utility than there is trying to work out the purpose of each class implementation form the point of view of the abstract class. What does it matter? It doesn't.
Open source software doesn't suffer the tragedy of the commons precisely because the utility contributors gain can only be upwardly increased by people using it, and overconsumption of the software has very little effect on the utility of the producers. In the original case of the commons, overconsumption of grazing lands decreases the utility of each individual grazer, but if anything the opposite effect happens in OSS. Assuming one can brush off idiotic bug reports and I'll-advised pull requests, that is. But because contribution is not binding when the cost of contribution exceeds the utility gained, the person switches their time preferences to differnt things.
Incidentally, the evils of central planning are precisely because people try and 'solve' the problem of marketplaces, when there is no problem to solve, only exchanges to observe.
As someone who does come from a hard science/engineering background I tend to find that every time someone tries to explain economics to me the more alarming it sounds.
The tools of economics provide a dim light through the fog of chaotic crowd interactions. It's not suppose to be a blueprint.
It's well known but it still has a circular definition. Why do people spend $X on goods? Because it provides $X worth of utility. Why does it provide $X worth of utility? Because they spent $X on it.
Ultimately it therefore ends up being rather meaningless. It's also non-disprovable which makes it non-scientific.
Sadly he seemed to have managed to talk himself into circles trying to argue that labor was the source of value, and therefore capitalists were labor exploiters.
Because of this he ended up mixing use and exchange value when talking about production machines, to try to show that said machines didn't provide any more value output than was initially put into making it.
I suspect that once one correct for that, by for example making machines into labor amplifiers, Marx's writings have, given the circumstances and time it was written, some serious insights into the workings, and pitfalls, of capitalism (most people barely touch volume 1, but volume 3, published by Engels after Marx death, holds a stern warning about the effects of finance capital).
Using that as a springboard to bootstrap some other kind of society may well be futile however.
Well, of course they're different.
It does seem that Neoclassical economics is fundamentally structured upon a series of misdirections that benefit capitalists:
* Blurring use and exchange value. * The non-existence of money in neoclassical models. * The non-existence of banks or other credit intermediaries in neoclassical models. * Even the non-existence of profit in neoclassical models ("perfect competition")
Pretty much any kind of element of an economic model that could be used to analyze the source of profit, in fact, is just handwaved away under the heading "assumptions".
Your first mistake here is to try and assign a '$X worth of utility' - utility is not a unit which can be expressed in another unit. Someone can spend $x on something which gives them utility, but equally they can go for a hike in the mountains which provides greater utility to that individual than the object bought for $x, yet the hike costs has no $ value and cannot be bought nor sold.
So the circular argument criticism is only circular because you're forcing it to be circular.
Neoclassical economics definitely does make a pretense of being scientific. As does Marxism.
For example, let's say you've always wanted to go skydiving. You pay $500 and go skydiving. It's a great value to you. But when you get back to the ground, would you pay $500 to do it again? Probably not, since the utility of going skydiving has dropped for you (because you crossed it off your bucket list), even though the price has stayed the same.
Now imagine the price for skydiving has dropped to $25. You might go again. Heck, you might go every day for a week. But after a while, the novelty wears off, and you won't go anymore. Even if the price drops to free.
In fact, if someone was forcing you to go skydiving every day, you might even PAY to do something else instead.
On the other hand, maybe you need to skydive every day because you are trying to qualify as a stuntman. That means the utility of being qualified is greater than the negative utility of "boredom from too much skydiving".
Or maybe you NEVER go skydiving because you went base jumping several years prior, and skydiving seems similar enough that it's just not worth $500 to you.
Trying to representing these trade-offs with only money is confusing and/or impossible. Try to work out the "boredom" example using only money: Someone can know they get more utility from X than Y, but not be able to express prices for X and Y.
That is just a sometimes useful abstraction and money does allow for quantizations of value. But utility is a broader idea. It means 'subjective value' and can differ from person to person, time and circumstances.
There is no unit of value, there are only personal preferences and those preferences can be comparable, but not necessarily quantifiable. How many 'utils' of value do you gain if you exchange $3 for a sandwich when hungry?
And it's perfectly scientifically sound idea. It's just too complex for us to solve very accurately. (Like a lot of important problems in science.)
I did much less development than I did before Github came along...so I don't know if Github (which of course itself depended on the existence of other OSS and infrastructure advancements) is a cause, rather than just a correlation with my career trajectory. But I can't imagine having the same willingness to work on OSS if the status quo were SourceForge (even before its current bloatware incarnation) or even Google Code, which was a pretty good service.
It's not that free software back then was less free than it is on Github. It's just that the barriers for dilettante developers to take that first step into OSS are lowered in very important ways. Without knowing anything about a library other than the part of its API that I use, I can make what I consider to be useful addition to the library and have immediate and reasonable confidence, given a decent test suite, that I didn't break the countless other moving parts in a library. Sure, test suites have existed just fine long before Github, but at least I can _see_ that a library has a test suite before even downloading it.
My involvement with OSS could be done there, but I'm literally a short commit message and a couple of button-clicks from giving my work to the library's maintainer as a pull request. If it were any harder than that, I know I would put that pull request in my nice-to-do-someday pile of things and then completely forget about it.
On the mainatiner's side, not only does Github provide a pretty nice notification system and workflow for them to organize, browse, and comment on incoming pull requests...but the way continuous integration services have decided to hook into Github means that the maintainer literally has to do nothing to have a first-pass confidence that I didn't break the library in all the ways anticipated by the test suite. This obviously does not ensure that the feature addition is not-shit, or even completely safe...but the maintainer has more time and energy to evaluate those important questions.
And then on top of it, if the maintainer accepts the pull request -- within a span of minutes if not seconds, anyone in the world who then downloads the library or updates their copy has what I added to the library, without me having to do a single thing beyond submitting the pull request. Even besides the hope that rising tides lifts all ships...the instant gratification from the OSS process alone could be enough for otherwise disinterested developers.
Essentially, open source provides a mechanism to share the cost of the long-term maintenance of a project. By contributing back, you can leverage the resources of the community to ensure that your new features continue to work against new and improved versions of the software, without requiring future resources to re-implement and update any changes made.
Because they get paid a salary to do so. Because commercial companies poured millions of dollars into the OSS projects. Specifically, the projects that people actually think about when they talk about successful Open Source Software - Linux, Firefox, Chrome, etc.
>as well as the ability to leverage involvement into actual cash like well paid consultant gigs, books or successful blogs.
That only applies if your project is already successful. And if its already successful, chances are high that you're already drawing a decent salary. And, books... Err.. Do you have any idea how little money authors make on those?
http://www.techrepublic.com/article/for-50-percent-of-develo...
The vast vast majority of Open Source projects are just that - Source Code. No chance of money. No chance of glory. An itch was scratched. Someone just pasted a bunch of source code onto the Internet. And there is nothing wrong with that whatsoever. However, painting the entire OSS landscape with the success of the extreme minority distorts the picture.
An interesting, tangential point is that OSS is probably the most pure meritocratic way to build a reputation -- it does not rely on gaining access to a powerful network (which relies on gaining access to a slightly less powerful network, which relies on ...) or other variations of more-or-less controlled luck.
A real life example might be a small town in an emerging country [1] where retailers band together and invest in building a road to the big city nearby. This greatly increases their revenue by bringing in a lot of traffic that wouldn't otherwise have bothered, even if the ROI on the road, if measured purely in terms of money in money directly out, is "negative".
[1] I read about this happening in regions of Iraq. I saw it in person in the state of Maharashtra in India, where Godrej is building roads and power plants in the region around its factories near Pune.
If the trend continues until nearly everything is free+open then there may be less and less work which is considered "value added".
Or they could have talked about how open sourcing is part of a race to the bottom. They mention some SQL-like databases, which is a tech that started out as proprietary. How much tech starts immediately as open source?
1. This is a cambrian explosion of open-source infrastructure tools
2. This is the dark ages of open-source UI
I think it's important to be aware that on this front the Open Source movement has completely stalled. When I was in College, I used Abiword, Evolution, and Rhythmbox. Now those things have been replaced for me with Google Docs, Gmail, and Amazon Music.
I love the web, it's incomparable for convenience, data protection, and access on multiple devices. But it is fundamentally at odds with Open Source software. Back in the day of desktop software, you could throw a tarball on the web and people could use your stuff without any real handholding. In order for people to use a web service you write, you have to maintain a server instance. It's just not worth it for me, random open source web service author, to maintain a server for millions of you, random open source software users. It's time and it's money for very little direct benefit to me.
And on app stores, it's pretty bad too. Maintaining a reviewed and updated app in an app store is a huge job, and not something a casual contributor to an open source GUI is going to want to do, just to have access to their branch of an app. And I'm not even sure Apple will let you submit an identical app with just one little feature changed.
This will change... the endgame is something like Ethereum, and the currentgame is maybe something like Heroku Button, but there is still a lot of infrastructure to build to make it as easy to maintain an open source web app as it is to maintain a desktop app.
And in the meantime, Open Source UI is just going to languish a bit. That's OK, it's just where we are in the history of the web. But it's going to get a lot better in the not too distant future.
I don't agree with you because the browser is ubiquitous and the web is open and based on standards. And you don't actually need a server to do interesting things in the browser. Quite the contrary, because of modern browsers, a majority of people with a PC have access to a potent development environment. And then the difference between the web and app stores is like night and day.
We can't have an open source web app workflow until the entire develop/deploy workflow can be forked with a couple button presses. As long as there's one single part of that chain that needs to be done by hand, we won't have a rich pool of open source web software users.
You're right though that if you're willing to use Desktop software there's still a bunch of good software out there. I'm just not interested in desktop software anymore.
Then there's the question of "customer acquisition". Low-commitment people don't seek out the best software, they have a range of options thrown at them and they pick one. Hence the current business model for startups: take people's personal data and sell it so you can spend the money on advertising your free service to them.
"You can't compete with free" is a double-edged sword: libre non-exploitative software has to compete with "free to play" software that charges with your privacy and exposure to adverts.
Another factor is that much of the best open-source software is similar to the best Unix tools: they do their one job very well. Being able to install all of them at your leisure is not much more advantageous than having grep, curl, and awk installed but no text files to work with or knowledge about how to use pipes.
I was trying to think of closed-source, focused software that is most definitely better than their open-source rivals. Not counting full-featured suites such as MS Office/Google Drive versus LibreOffice, or GIMP vs Photoshop...the first thing I can think of is ABBYY FineReader for OCR, which I've never used but the results of which seem to blow Tesseract out of the water, even though Tesseract is very good. But perhaps its niche is not broad enough to attract the level of open-source development in the same way that software and database architecture has.
Has there been any empirical work on assessing it in recent times? Maybe post 2010?
For almost everything that requires other expertise the innovation is still done in closed source software.
Stuff like photo editing (LightRoom, Photoshop, Capture One), video editing (Premiere, Flame, DS), high end rendering (Renderman, 3Delight), CAD (Catia, Solidworks, NX), embedded systems (almost anything in cars, airplanes, appliances), and a million other areas are all still as closed as can be, and open source isn't even competing, much less innovating.
And it may even be technically true for the examples I mentioned. But in those areas, for all practical purposes, research from academic and corporate research doesn't actually get used until it's implemented in closed source software. At the annual Siggraph conference a good portion of the papers come from companies like Pixar and Autodesk, and get implemented in their products. Outside of that, it's pretty rare for the academic stuff to go much beyond a cool demo, unless one of the big graphics companies thinks they can use it, at which time they buy up the rights to it.
Using graphics as an example again, imagine a research paper explaining a new rendering algorithm that can render in half the time of the next best algorithm. As great as it may be, it's not very useful without the rest of the software infrastructure that comes along with a production rendering system, like Renderman, 3Delight, or Maya. So even if the academic researchers release an implementation as open source, it's unlikely to be used in production until it's incorporated into one of the big closed source systems.
I suppose pay walls could be a contributing factor, but in a lot of areas the open source solutions are so far behind the closed source systems that inaccessible bleeding edge research isn't biggest problem.
Far as your example, I agree that algorithm support can't help without good software going along with it. That's probably true for many use cases with examples you cited being areas where proprietary tools and ecosystem dominate. Leads to some potential counters:
1. It might help in the many things that OSS is doing. That's a huge list that includes desktops, phones, servers, and a few major apps.
2. It might help in commercial sector if their developers see the cutting edge approaches. It might be obvious where rendering advances are but most stuff is scattered: finding it took lots of searching with my keyword, kung-fu. Might be affecting discovery in proprietary side, too. I know many of them keep inventing solutions weaker than even well-cited academic stuff. Aside from patents, gotta wonder why they didn't go with better approach if it seemed like a good fit. Likely explanation: didn't know it existed.
For instance, it's commonly believed that signal processing can only use a commercial DSP or graphics card with nothing delivered aside from that. People wanting an open alternative don't even know where to start that's worth anything. I heard and believed that too till I landed on this bad boy:
https://www.wikiwand.com/en/Asynchronous_array_of_simple_pro...
The architecture is described in enough detail to be clonable. It might even be licensable with open-source allowed. It should handle plenty of workloads and supports custom accelerators for common things. After the mask costs, it would have great price vs performance vs energy usage. Yet, with 8 years of research, I just now came across this despite the first deliverable being in 2005. Might be a dead end and might turn into something if pursued. Yet, I'm sure it had potential to benefit commercial and OSS projects whose design approach accomplished far less had they known about this.
So, more full and legacy systems certainly have lots of momentum. You can't just have a piece of a system. I think even they might be affected by the problem I describe, though, given they're as sub-optimal vs cutting edge as OSS often are. So, easier movement of these ideas might benefit both.
Most OSS just sucks compared to best proprietary stuff. It still can't do better. So, proprietary tools are the ones that OSS is still judged by. Certain categories, esp in infrastructure, have OSS kicking serious butt. The dynamic works the opposite way. Even stronger are the hybrids of proprietary and OSS that make the argument less meaningful. Many of us have predicted that hybrids would smash both with some evidence of this in cloud sector. OSS still isn't in that dominant position in general, though.
My point is that despite the growth, there are still many, many areas where the open source offerings are pretty lame and noncompetitive (if they even exist). Open source has never been very competitive in any of the areas I mentioned.
All together, leads me to wonder whether most OSS should be considered proprietary development or how much community-driven efforts actually accomplish.
The software engineer occupation is a highly specialized role. A software engineer alone does not have the ability to survive, hunt food, or build shelter. Instead the software engineer must trade his skill for the services of other experts.
In short, a software engineer who only gives away his work to the open source community basically can't pay the bills. He needs to sell his work. Although open source is built off of the leisure time of programmers that leisure time must be bought and paid for by an external factor. More often than not, that external factor is Closed source.
Ironically, open source only exists because of closed sourced apps or web services.
'Proprietary licenses' should expire and become open just like copyright is supposed to.
I also think this is why jobs involving OSS development are increasingly attractive to candidates. Do you want to spend your time building what amounts to a one-off solution to duck tape some wonky IT-department components, or do you want to work on well-engineered projects that are generalized solutions?
So I think closed source software is basically the last layer that integrates open source software. Which ever side you view as bringing in the sales and money is really just a matter of perspective.
To use a couple economists' terms, positive externalities are generated when code is open-sourced, and the beneficiaries are the companies contributing, because together they're making something greater than any one firm could build alone.
Our software is orders of magnitude better simply for having been opened to hundreds of pairs of eyes. The QA is intense.
Engineers at closed-source deep-learning startups chafe under their gag orders, especially since so much of what's closed quickly becomes outdated. Lack of transparency can easily hide lack of competence.
I don't say it's not progress. But if we end up doing all the same things the same way, the prize is not that bright.
I'm not sure, "closely-guarded" is the term I would use.
Open source is a gift economy, like science without patents!
Where is Mises and Hayek ?
There could be a comparison to communism with the few that have a foundation, community-driven development, and maybe community voting on what happens with it. Closest to how people would picture it. So, which major FOSS projects are done like that?
"a theory or system of social organization in which all property is owned by the community and each person contributes and receives according to their ability and needs."
I suppose there is some superficial resemblance, but I consider communism a means of doling out limited resources in an equitable fashion. So instead of having individual ownership, the community owns it and manages it based on need. Instead of being free to act in your own best interest, you are obliged to contribute to the level of your ability.
With FLOSS, we are simply removing the concept that a piece of software is a limited resource. There is no group ownership -- individuals still own their copyright. It's just that under the rules of the licensing, ownership is irrelevant.
For example, I no longer sign inventions agreements at work. Instead, I tell my employers that all outside work I do will be under the GPL (or AGPL). I will even assign copyright over to them as long as the code is licensed as GPL. They can own it. I don't care because it makes no difference who owns it.
In a communist society, ownership is still quite important. It dictates who will decide what I need. If I owned the resource, I could take as much as I wanted (or give away or sell it if I wanted). With community ownership, the community decides. With FLOSS, if I have a copy of the software, it just doesn't matter who owns the copyright. I have the same rights and obligations either way.
Another big difference is the reception of the item. In a communist society, I have a right to the things I need and an obligation to contribute to my ability. With FLOSS, nobody must give me anything. If they do happen to give something to me, then I can do what I want with it (up to the limit of impinging other people's freedoms in some cases). But I don't have to do anything.
This is often a big misunderstanding the neophytes to free software have. They often demand bug fixes, or they demand to be given software for free. You just don't have that right, even if it is something you need. Similarly, if I make a change to any software, I needn't give that change to anyone. I am free to do so, but I have no obligation.
In summation, I don't think this is communism at all. It is simply leveraging the fact that existing software is not a limited resource. It establishes a set of community rules that optimises the freedom of the members of the community, not the satisfaction of their needs. Importantly, it does not establish any rights or obligations.
The last bit is tricky as you have obligations to offer source code if you choose to distribute, but the "if" part is important. Similarly, advocates feel that there is a moral imperative to act in a way that preserves software freedom for downstream users, but nobody says that those users have a "right" to the software (in the way that a person has a right to a food in communist society).
IMO, FOSS is more leaning to the socialist/anarchist/libertarian side of the political spectrum with decentralization, resources sharing and distribution and disdain for authority at its core.