Open Source is not dead: On the recent demise of RethinkDB
deepstream.io
deepstream.io
I looked into their business model a bit more... and it was like, super difficult to figure out how to actually pay them for anything. Seems like having a push-button managed DB option on AWS/Google Cloud/Generic would've been awesome for them? The company just didn't seem like it was set up to make money even though people liked the product.
I know, hindsight and everything... but without going to a page linked from a random github issue that I found via Google I couldn't even find a page talking about pricing. To build a business you have to not only make something people love... you also have to let them pay you for it.
No pun intended, but I wrote a blog post about this a couple weeks ago called Funding Open Source by Rethinking the Assumption at https://supportedsource.org/blog/funding-open-source-by-reth...
By rethinking the assumption (of a free price) we can keep a lot of the benefits of open source, but also have a model that just might solve the funding problem, by having end user companies pay. It's what I'm calling Supported Source.
If you run a project and are interested, get in touch.
The alternatives are two things: dual-licensing or just charging for the open-source software. These are fully open-source where they have no risks of proprietary software but risk loosing profit due to freeloaders or knockoffs. The latter option, charging for OSS stuff, threw me as I thought it wasn't allowed per some of the license but an OpenSUSE employee pointed out it's basically what they do. Some others do, too, including one person whose company does custom jobs for other firms plus GPL's anything they make. You basically make it available for people to grab but just keep charging corporations for it anyway. They don't know the difference as they're paying for specific benefits.
What's your thoughts on these OSS options?
EDIT: Wait, yeah it was Quinn with Fair Source. I've been too busy to get back to him but I see all of us having a more extended discussion on these things in future when things stabilize for me. :)
Interestingly enough, similar models seemed to be more common back in the dark ages of computing before terms like Open Source and Free Software were really a thing, particularly on minicomputers and mainframes. It made its way to the "microcomputer" world in more limited fashion, then sort of slowly died out. But I think there may still be something to the notion of a license that, while not remotely meeting the definition of "free software," still gives you access to the source.
I'll be honest, though; I think FOSS advocates have a tendency to (perhaps unconsciously) downplay how important "free as in beer" has been for the adoption of a lot of major software projects. An internet in which Apache, MySQL, PHP, and other major free software projects were released with commercial "source included" licenses would probably look very different.
I have my own thoughts about RethinkDB's failure as a commercial entity, but despite having actually been inside, I have pretty limited insider knowledge. Perhaps we should have been trying to push an "enterprise edition" with guaranteed support much earlier, following the "charging for OSS stuff" line you mentioned; as others have said, maybe we just didn't have enough avenues through which to give us money. (It's hard not to notice that MariaDB's .com site is essentially page after page of ways to give them money, for instance.)
I think a combo of paid with source and closed-source extensions can still get lots of success. We're already seeing it with enterprise editions for features that basically only enterprises use. That keeps them from crippling the main, OSS product which costs less. I also envision giving free licenses to people that contribute lots of code, docs, testing, etc. One way is to simply not charge businesses below a certain revenue amount or only to charge a support fee tied to cost of delivering support. As they grow, so does the software house like in the royalty model.
Profit? Sure.
Longevity? Not even close.
Things like the Spice3 circuit simulator and the Magic VLSI editor can continue to exist and run 30+ years later solely because they are open source.
However, is MCP actually running directly on modern hardware or is it being emulated through multiple layers?
Of course, you can use my own argument against me that the ports of Spice 3 and Magic to modern hardware are hardly the same program as 30 years ago.
The same reason for a lot of what's in open source. See PHK's A Generation Lost in the Bazaar for some truly unimpressive stuff.
"However, is MCP actually running directly on modern hardware or is it being emulated through multiple layers?"
x86 is emulated on a custom RISC machine. So does it matter when saying whether x86 is a proprietary ISA that still dominates desktops? MCP is proprietary software whose apps keep running for decades regardless of what hardware they're running on. Like other mainframes, the AS/400, whatever. I hope you're not looking for loopholes around the fact that proprietary software can last decades if it's mission critical, supported, and hard to move off of. It can and does in quite a few cases.
"Of course, you can use my own argument against me that the ports of Spice 3 and Magic to modern hardware are hardly the same program as 30 years ago."
I can't since I don't know enough about them. The mainframe and futureproof markets for software try to maintain backward compatibility with stuff you don't change. So, the part the app runs on is probably the same with who knows what underneath. Sometimes that part is ported, emulated, virtualized, etc. Varies. API and software above stays the same.
Now, (a) and (b) work for some use cases, but I'd argue that it's limited, especially usecase (a) because it still allows any 'customer' to use your software for commercial work without contributing anything back other than attribution. What would be nice is some kind of ecosystem around a license I mentioned above, where predefined percentages (that ideally can be set by each license owner) trickle upstream to support their project. Like an appstore for OSS libraries and software where you can register your software and the licenses you offer - if your app is free you don't need to pay anyone upstream, otherwise you keep (1 - x - y - z) if you used other OSS with this model that have predefined cuts x, y and z.
https://www.gnu.org/philosophy/selling.en.html
You can do that without a copyleft for BSD types to keep your proprietary extensions secret & harder to copy. Doing it for either is a good way to make profit or just raise development funds.
Regarding percentages, I think that's an interesting idea but worry about scale. Systems nowdays are getting big and full of components. More complex, but useful, software might get less market share due to penalty of high price. Maybe institute a cap for upper limit of a single component's license. Might be specific to the component or in general. Personally, Im leaning towards simple, annual licenses integrator pays per component. Might get each free for a year per startup so they can make some money. Crowdfunding, nonprofits, or corporate sponsors could help larger ones get started.
I don't quite follow I have to say. The distribution part of software can be rather trivial for niche markets where there aren't any big vendors around - with GPL I'd have to assume that I can only sell once and then the customer will just redistribute it - whether free or not - to whoever wants it, at the very least within his/her organisation.
> Systems nowdays are getting big and full of components.
I think it would actually encourage to have a nice long thought about what dependencies you want to introduce. Market would probably sort that out rather well, and things like the node.js dependency hell would not happen in such an environment. You'll first start with a language that comes with a rich standard library in the first place.
Edit:
> You can do that without a copyleft for BSD types to keep your proprietary extensions secret & harder to copy. Doing it for either is a good way to make profit or just raise development funds.
Yes, to me also this seems to be the best way forward with commercialising OSS without going the service route.
Many enterprises won't buy something at all unless there's a price attached presented by a company they think will be around a while and help them with any customization. This market is less likely to trust a competitor to Windows Server or Exchange if it's being offered for $50. ;) Plus you can point out it's better to buy the real thing supported by developers that invented it instead of cheap knockoffs. Call them cheap knockoffs.
So, there's some risk but it can be done. Best model is having some proprietary addons, though.
"I think it would actually encourage to have a nice long thought about what dependencies you want to introduce. "
I thought about this as I wrote it. The reason I didn't add it is because you'd have two, competing solutions: FOSS community that mixed and merged no matter the size; shared source community where things cost more as complexity went up. I figured the latter would be largely boycotted if former was available. The former has so far been fine for vast majority of freeloaders. I doubt that would change. The latter could become a niche market with low complexity a differentiator, though.
"Yes, to me also this seems to be the best way forward with commercialising OSS without going the service route."
It's what I'm going to do. If money rolls in enough, I'll also submit select improvements and/or donations back to the source projects. Just on principle.
would be proprietary software you don't have to reverse engineer
>[AGPL + another] still allows any 'customer' to use your software for commercial work without contributing anything back other than attribution
How so?
AGPL doesn't have any mention of commercialization - you can use it as an end user in any context for free, but you will have the same issues with selling derivative works due to copyleft. For software that's actually just software and not a service, say media production, it means that *GPL software will forever remain tools for hobbyists.
I continue to be amazed at the number of startups where I have to google around for pricing. (This is excepting those folks who want you to call for a quote, of course.)
We find people who start with community edition will eventually move to enterprise once they establish the value it brings...and then eventually they will pay for the enterprise license. Yes, we have tons of people using the enterprise edition without paying... and that's ok. They'll come around too eventually once they see the value.
If we can build a business going full open source, you can too. Don't let the commercial failure of RethinkDB scare you otherwise.
We don't have any users who contribute to the codebase. But there are a few elite users out there who build their own components for debugging purposes. This lets them send us very precise bug reports that get acted on quickly.
So, all in all, we get a little bit of benefit from being mostly open source.
For me, the value in open source comes from being able to learn the depths of a code base, or to see how a pattern was applied to solve a type of issue that I may be trying to solve in an unrelated technology. If I find a bug I will submit a PR where I can, if I understand the code base well enough.
That said, I think it's a Utopian ideal that going open source means that people will flock to contribute to a project. Companies like Microsoft are learning this the hard way I feel. Have a look at their .net core repos. The vast majority of users are simply "throwing issues over the wall" without trying to contribute any fixes or discussion.
I would love to live in a world where everyone attempted to produce more than they consume, but we just aren't there.
EDIT: P.S. I love Couchbase!
We're not going to get there, either. The number of users with the time available to dig in any depth into an issue they find is always going to be dwarfed by the total number of users contributing issues. That number in turn is dwarfed by the number of users who don't even go that far, and just work around problems they find (probably by silently switching to an alternative product). This isn't a moral issue, it's simple economics.
I'm going to modify that a bit to say it's mostly economics but also barrier to entry. Wirth's Oberon etc work was designed to be easy to understand from language to architecture to tools. Students with little experience regularly ported it, redid it, and so on. I've seen similar in FOSS projects well-designed for easy understanding. I think many of these popular projects were not designed that way to the point the real and incidental (i.e. implementation) complexity is to high for most to justify time in. I know it is for me for several, major projects.
So, we need to remember that. More FOSS being designed for contributors as much as users might increase number of contributions.
Usually you have to fight for the right to contribute back, which in big companies means lots of fun with upper management layers and legal department before you get the green light to do that little pull request.
So in my career I have seen lots of open source being used at Fortune 500, sometimes with internal forks of bug fixes and zero contributions being done back.
Yes it is.
But once you start making serious changes to the source-code, it's probably less painful than trying to deal with all the merge conflicts from your own fixes every time you upgrade to a new version. At some point it's just easier to push all of that stuff to the common version.
What matters are all those companies that no one heard of, or whose business has nothing to do with producing software.
> a world where everyone attempted to produce more than they consume
What I do is try to write easily reproducible bug reports. However, if I were to even attempt to contribute any code I would have to make a major effort. There are few problems on a foreign code base that can be solved in an hour or a few hours.
This is why I ended up using Intellij instead of an open source editor: I once submitted an issue on Github and the response was basically "where is your pull request". I understand the developer(s), but sorry, I am just not able to do that. Imagine that a sizable number of OS projects only deal with your reports if you are an actual contributor. Sounds like a good idea? But how would any person even manage to be a contributor to even half the projects that I am using? Anyway, back to the IDE, I pay Jetbrains and give them great bug reports and they fix most issues.
I think it is completely normal, just from a statistics and system modeling perspective, that only a tiny number of developers contribute to a project, and that those mostly are people for whom that particular project is a core issue. It is just not feasible any other way.
--
Here is a crazy idea (only half serious, meant mostly to bring a new direction into the debate - so no need to point out potential problems, it is not a suggestion made to be implemented).
Imagine we had a worldwide organization that you pay a certain amount to. Several levels, flat fee but with levels.
The organization watches what "free stuff" you use. Software, code, anything that's bits and bytes, could easily be extended to 3D printer models photos movies etc.
Then they distribute the money they got from the (semi)flat tax to the makers of the various things.
The different payment levels indicate a) amount of use, someone using only a little bit of free stuff vs. someone who uses tons of it, b) level of sophistication, so use of the left-pad npm package and similar stuff is a different level than someone using Blender, for example.
Actually, such organizations do exist. In Germany there is one that collects fees from copy shops and distributes it to right holders of printed works ("VG WORT"), and there is another one collecting fees for music and film ("Gema"). Lots of problems, of course, just saying that the idea itself is not all that outlandish and attempts at partial implementation for specific countries and sectors already exist.
Again, just to show another option compared to no solution and continuing what we do, lots of problems of course, like how do you prevent someone from inflating their usage count, privacy, etc.
How about a limited implementation of something like this? We need some way for contributors of "free stuff" to be able to live a life. I think it is bad to a) accept and b) rely on such people making a living somewhere else. It is too random.
I am not a "socialist". It is entirely "capitalistic" to expect that there should be a reward system for actually useful contributions (as measured by usage). I just think our 19th century system is very badly suited to handle the new digital world, from copyrights to open source.
The problem of individual payments also is that the per-payment cost is too high - and I also mean the psychological cost. Imagine you had to buy a car screw by screw instead of as a whole package, being asked for every single tiny piece "how much are you willing to pay or contribute?"
It could also be a potential solution for this problem also on the frontpage right now, because those people could go from the official job market where they are noz wanted into the system I describe: https://news.ycombinator.com/item?id=12724104
---
I also want to make a connection to the Universal Basic Income discussion. I am skeptical of the "pure" idea, I think we are still far away from allowing universal leisure and I don't believe that most people will voluntarily do something useful. But how about the above idea can be connected? If you do something useful that's all that's needed, but you have much more freedom to decide what that is. Mostly, "distributing free stuff" now works as a business idea!
Just had this experience - reported something in the grails slack channel, and one of their engineers hit me up, we exchanged info... he wasn't quite able to replicate my exact issue (but I wasn't able to either, reliably), but based on the stacktrace, he found the likely source, and patched it. No pull request, no forms to fill out, etc. Was a pleasant experience. :)
Without an actual reward-connection though this comes down to luck on many levels. I mean it's nice for me when somebody gets paid from a completely unrelated source and yet is willing to help me, but that's not a very reliable thing to rely on a population level, or let's say I think the success rate can be greatly improved by establishing a reward connection.
SPECIFICALLY, this was in relation to a Jetbrains IDE - IntelliJ - the paid version has Grails support. Sorry that wasn't clear.
Doesn't that mean you do produce more than you consume, just in different directions?
>I once submitted an issue on Github and the response was basically "where is your pull request".
As a clear up from someone who has seen this kind of thing in Github often. Most of the time it's not meant as a "will not fix", usually I see maintainers use it as a "You are welcome to give it a shot if you want!". Some people need that express permission to put in the time to actually do it.
--
As for the rest of your post it went way over my head, caffeine hasn't quite kicked in yet.
The solution as I see it is Supported Source (https://supportedsource.org/)
This is what FOSS actually made in the last decade, moved many that would never pay for software from pirating it to just downloading whatever they can find in open source repositories.
But the developers still don't get anything back.
Which isn't the case in many countries.
Similar reason why companies like Netflix exist. Imagine each movie would try to sell itself, they would get their own webshop, one per movie... and similar to why subscriptions exist and many other bundling arrangements. Making payment decisions comes at a psychological cost, but just because it's "psychological" and your brain doesn't pay some other part of itself with coin or credit card doesn't mean it's not a cost economics cares about. Cost in economics is not just money.
Another example is Twitch. In addition to giving mostly gamers (but they also have programmers streaming and other non-gaming activities) a technical platform to stream content from home they also give them a monetization opportunity. While you can subscribe to streams individually, which is of course not what I suggested, they now have a program where you can subscribe to a handful of channels for free if you have an Amazon Prime subscription (http://www.theverge.com/2016/9/30/13125824/twitch-prime-amaz...). So this is a partial example of this "bundling" of small individual items to increase the overall spending of money compared to having to make individual purchases.
I see this as an opportunity for achieving something similar for people who contribute code, there could be a business opportunity to bundle a large number of those offerings and in turn offer the programmers a way to monetization, to get paid out of the subscription pool. It's much more difficult though because the audience on bot sides, those offering as well as those buying, is much more discriminating, and the situation already is far more fragmented. For things like gaming streams it never was (very fragmented) to begin with. Imagine Github had a subscription and would pay out to projects according to use (stared, downloads, whatever).
Also, it seems that the community versions are drops from the commercial product - that generally causes lack of community.
Users having the ability to edit with the code, and provide detailed bug reports is a massive benefit - and is probably a reason that these customers are using the product.
(ex couchbase employee)
I work on a proprietary product (SaaS + on-prem models), but this is also a very strong consideration for us in choosing basically anything we use (whether part of the product or part of our infrastructure).
If you're a small shop and not open source, unless you're literally the only option (or you're trivially replaced by a dozen other products), you're basically ruled out, unfortunately. This includes SaaS stuff.
It's not that we don't have the money, it's we don't have the risk tolerance. If you disappear, or get purchased and stop offering your product, or decide not to fix our bugs because they're too niche to us, I don't want to be hung out to dry.
Lots of examples. Convergent's CTOS, OpenVMS temporarily, QNX's "open" source, most desktop OS's, many commercial compilers, and recently FoundationDB I had hopes for. I simply don't use something commercial unless I can export the data out of it easily to an OSS alternative that I have ready-to-go. Also need to regularly export and test that data as the closures or licensing changes sometimes happen without warning.
Open source or not didn't even come into play, They had great interest, devs liked them and they figured out how to do meetups without much budget. Devs I knew said they liked them team but shortly into it it was clear they were hoping a revenue stream would stick and went forward on that would just happen. Strange one really.
Open source in practice, however, means 99% of users never contribute back anything. These developers just end up being exploited (particularly by well funded companies) by working for free, building something valuable, fixing people's bugs, but not being paid for their time.
That's what I want to change with Supported Source, an alternative model to open source. Open source developers: don't give your code away for free any longer! If companies are willing to use your code, they should pay you (it can still be given for free to small companies or individuals) Check it out at https://supportedsource.org/
For example, usually with the professional version of compilers the source code was also there, just not on the student/hobby versions.
Just recently I got hold of an old Turbo Pascal 5.5. brochure where Borland explicitly referred the availability of the source code on it.
Yes, open source brings commercial benefits to a company over closed source but it's not as big of a differentiator because it's quite pervasive now (which is good!)
Both are based on paid consulting/support and try to monetize their open source software (RethinkDB and Horizon on one hand, deepstream.io on the other) through hosted versions, the Horizon Cloud (beta) and the yet to be released deepstreamhub.com. I really hope this works out for deepstream.io, maybe the focus on PaaS makes the difference.
As long as the open core is still a quality product with good support and documentation I don't see the problem. Nearly every other open source database seems to be built this way and it seems to be working fairly well for GitLab (who are also pretty open about their development of the EE although the EE itself is not open source).
IMHO this is the main problem with open-core and it is a really tough decision you have to make - is multi-process a standard feature or not? Because ultimately I would have guessed it is a fundamental feature of a job scheduling system (assuming long-running tasks) and one could say it is even at the core of a scheduling system.
Don't take this the wrong way, I applaud everybody who finds a way to develop meaningful FOSS (and a reliable working background job scheduling system is not a walk in the park) and getting paid for it, so good luck with it!
1. My paid features are all built on the core APIs that are open source. Almost every feature I sell has an OSS variant that someone has written. If you want to plug in a dozen different OSS libraries, integrate and test them together, you can get most of what I sell for $0. You will get varying levels of support and yearly upgrades of that stack will take days or weeks to validate. My customers recognize that building and maintaining any system like Sidekiq is non-trivial. You can spend a week or more annually maintaining your own Sidekiq "distro" or you can pay me to do it. So far, many hundreds of companies have opted for the latter.
2. Regarding multi-process specifically, each Sidekiq process is multi-threaded and will execute (by default) 25 jobs concurrently. Multi-threading is the OSS feature and delivers a huge jump in efficiently versus the old single-threaded Ruby job processors (Resque, DJ). multi-process makes it easier to spread work across many cores with a single command but nothing is stopping users from running multiple Sidekiq processes, they just won't share memory and must be managed separately.
That said, your scenario is unusual. Most server-side jobs have plenty of I/O (think DB queries, memcached/redis calls, file I/O, etc), which allow the threads to context-switch; lots of I/O concurrency means lots of jobs get done quickly, even with one core.
Why were these projects able to build communities around their codebases, when RethinkDB was not? Part of it might have been large companies making investments that other companies could build off of. I think choosing an appropriate open source license, and Apache governance model was another big part.
Certainly, not all databases or data stores need to be open source. But there are still many successful ones out there.
1. Standalone open source companies like RethinkDB. These typically rely on secondary sources of income (e.g. support contracts, SaaS, etc) or eventually adopt an open core model (deriving a commercial product from the free open source project).
2. Major software companies that open source part of their work to push their commercial services (e.g. Microsoft, Apple and Google), or as an elaborate recruitment tool (e.g. Facebook).
3. "Independent" open source projects where maintainers are directly sponsored by outside companies or indirectly via foundations (e.g. Linux, Apache, Mozilla).
I think #2 is only workable at a certain scale (because open source work basically becomes a major marketing expense) and #3 typically requires several significant backers to guarantee long-term survival.
So looking at startups the only workable solution for open source seems to be #1 but because open source lies at the core of what the business does it's vital to generate a source of income other than the open source work itself (unless the open source work is de facto unusable for most users without a commercial license, e.g. Sencha ExtJS and Touch).
RethinkDB seems to have failed at becoming profitable. My understanding is that they tried very hard to maintain a "pure" open source ideology and therefore fully relied on support and training contracts -- which is directly at odds with their primary goal of making RethinkDB easy to learn and use.
I think RethinkDB's biggest failure was that as a product it didn't really find a market where it could become profitable. Based on their marketing, the main audience seems to have been startups. They were likely hoping that startups would give them a reputation and that reputation would translate to traction which in turn would translate into paid support, training and commercial licenses.
But even though they could have made it a lot easier to give them money (the support and training is hidden behind the "Services" section on their website and I only heard about the availability of a commercial license from RethinkDB users) I think that alone wouldn't have helped them because there just isn't much money they could have extracted from that particular market.
RethinkDB is unattractive to most companies outside the startup sphere that aren't tracking the new hotness. Streaming is a great selling point but there are very few companies who would be willing to trade for that benefit when it means a ton of drawbacks in other aspects. Horizon was acting as a competitor/successor to Firebase but even Firebase doesn't have much of a market.
People repeatedly point out SaaS as a possible source of income but creating and maintaining a SaaS product is an entirely separate can of worms from "merely" working on an open source database. Plus it brings with it a large set of financial risks and overhead that can be a major drag on resources -- all the while hinging directly on the success of the open source main product.
And SaaS also suffers from the same problem as offering support and training: SaaS is most useful when deploying and maintaining the same services on your own hardware (or cloud) would be hard or troublesome, but RethinkDB tried to position itself as easy to use and easy to learn.
For databases the only plan that seems to work for #1 is the open core model. True, it's not ideologically "pure" open source but as long as the community edition is a decent product on its own, there's no harm in providing enterprise services on top of it at a premium.
PostGres does not exist. Correct spellings are Postgres and PostgreSQL.