Update on InfluxDB Clustering, High Availability and Monetization
influxdata.com
influxdata.com
Several months ago it seemed clear that the team was overly optimistic, and it's just disappointing to see that now the clustering will be available only in a paid (minimum $400!) option or on their hosted service.
I understand the business considerations here, but it feels like a bait n' switch for all the people who evaluated/used InfluxDB in single-node operation as a temporary measure while giving the team ample time to work out the clustering kinks.
Lesson learned I guess... but dang what an expensive lesson.
The HA page with open source options we put up will be getting more detail over time: https://influxdata.com/high-availability/
We're comitted to continued innovation in the open source InfluxDB and see this as one of the avenues for ensuring that we can continue those open source contributions.
Also, there were some tentative thoughts about using InfluxDB as the main long-term storage backend for Prometheus, but that has become pretty much uninteresting now that clustering support (needed for LTS and durability) is basically cancelled for the open-source version.
Still, I guess I can understand that when you're a company, you need to focus on making money.
Also, there were some tentative thoughts about using InfluxDB as the main long-term storage backend for Prometheus, but that has become pretty much uninteresting now that clustering support (needed for LTS and durability) is basically cancelled for the open-source version.
True, but consider this: Given that InfluxDB (core) is Open Source, there's nothing to stop somebody else from coming along and building a version with clustering support. Whether that will happen or not is, of course, an open question.
Clustering is really really hard. My personal experience with that class of problem is that it took 3 years to get it production ready, and that was for a simpler problem than what needs to be solved here.
The chances of a 3rd party implementing this given that InfluxData plan to have a commercial version seem very low. I'd expect that most companies who want to stay open source will treat this as a standard database that needs client-side sharding and HA (which is the recommend approach - see https://influxdata.com/high-availability/).
We only use LevelDB for Prometheus's time series indexes, while using a custom-built storage engine for the actual series data. But Prometheus's local storage has always been built as single-node storage without clustering - very much on purpose actually, because we want monitoring and alerting to be decoupled and autonomous enough to not have any hard distributed storage dependencies (as that has a high risk of breaking down when shit hits the fan).
We definitely want to support full remote, distributed, and durable storage for Prometheus, but we still want local nodes to have enough autonomous storage for alerting and short-term inspection of data. Whether some of the technology of the local storage engine will be used for building a clustered storage remains to be seen, but it's likely.
I guess time will tell, but a bigger deal than the money to me is will the clustering actually work well? Clustered databases is an extremely difficult problem. Seen it get better in the years in things like MsSQL, but even with that sort of resources it took a long time (years) for the newer availability model to become stable.
At Stack Overflow we are still using OpenTSBD behind Bosun. But HBase sucks to manage if you don't have any other reason to be using the technology. So I see a lot of users interested in using InfluxDB as the backend (and some do, Bosun can talk to it) because it is easier to get started. But if you know you will have to scale up eventually, all the options right now are not appealing in TSDB land :-/ So if some $$ really does get a good TSDB that scales and reasonable to manage then great, but I'm skeptical.
In the meantime, the stuff we have going into the open source is going to make the standalone server (and thus anyone using open source options for HA or clustering) even more performant, stable, and scalable.
We use a relay (bosun.org/cmd/tsdbrelay) to send data to OpenTSDB and it also "tees" it to Bosun for indexing and another OpenTSDB cluster. So sending all of our data to an Influx cluster will be as simple as adding that as another endpoint to our relay since you have an OpenTSDB endpoint.
What open-source option for clustering would this be if the main InfluxDB open-source version won't have clustering anymore? A fork or something built on top of multiple standalone servers somehow?
I wasted a lot of time looking for a bulletproof time series db. I found nothing, each one of them have their own set of issues.
Lesson learnt: before deploying a new OSS, check if they have a credible plan to support the project. Otherwise skip.
I think HN in general is being a bit too cynical here. This is an OSS company trying to find a sustainable way to survive. There are going to be some tricky tradeoffs. "They should just charge $20/mo for us small guys!" is not a reasonable course of action.
Regardless of whether this particular pricing plan works out or not, I think it's a good idea for programmers to support other programmers trying to make money on OSS. We are all not served very well by the general attitude of "I heard about this technology last year, it should be $0 now".
We used to pay for this stuff - remember Sun? Things change.
I suspect what happened with InfluxData is that being venture funded, they see that the funding environment has changed and are racing towards revenue/cash-flow positive. And clustering as a premium feature is, apparently, the answer.
If clustering had been pegged as a premium feature up front then neither I, nor I suspect would anyone else, have a problem. But it wasn't, hence the sour taste.
It's fantastic that free software has increased so much in market share. I just wish that more companies would recognize that that only works if they actually pay for development of some projects in some way. That can be directly employing maintainers, contributing patches, sponsoring feature development.
Since that often doesn't suffice you sometimes end up with companies trying to make ends mean with "open-core" type models; even if they'd otherwise prefer not to go down that road.
I'm sure giving yet more things away would have saved them (that was sarcasm).
Software doesn't get written for free. That won't change. VCs can subsidize it for a while but eventually it has to be commercial.
It's also worth noting that FoundationDB was totally closed source and disappeared after the Apple acquisition.
That won't happen with InfluxDB. The open source standalone server code that we develop will continue to live on regardless of what happens with the company.
It's not rhetoric. I know people using it so it seems to me you're trying to spread FUD. There are still things we're working on, but we have many happy users that are operating at high scales.
I should also mention that almost all of them are doing this completely for free. Which is great. We want to subsidize our open source software
For that particular user, it is something else they're hitting. Without more info we have no way of knowing why they're hitting that error.
Either I'm lying about people using this thing at significant scale or that user is having some other problem. Put simply, TSM is well tested and works. I'm certain other bugs will come up but the comment you linked has nothing to do with TSM.
I should also note that we have users running Telegraf on thousands of hosts sending metrics to a standalone InfluxDB server.
I have no stake whether you get it together or flop, so from this perspective I will cherry-pick bugs because that is what matters to users of a database which should be boring, stable, reliable, efficient.
With the change of direction on clustering, and I am skeptic of that being done well any time soon given the storage engine history, you have a hard road ahead of you. Until/unless you get it together, there are better choices, especially for pay, and the "everything is fine" message is infuriating.
Also, despite being mean here I do hope you get it together. I just wish you'd reconsider the marketing approach and hold claims of a production release back a long time until the bugs are under control.
Maybe we can avoid clustering with some workaround, (sharding) but I feel tricked.
However, we think that the community will be better off if we can continue to contribute to the open source InfluxDB. We want to invest heavily in open and closed source software. Ultimately the open source community will benefit from an open InfluxDB
I'm guessing you also have customers and are trying to figure out how it is that you can get them to pay you money so that you yourself can create a long-term business and create better products over time. Right?
The clustering was (accidentally) bait and switched and led us to choose this product when we would have made other decisions.
One is that we would have a single point of failure.
Another issue is that we can't count on it being expandable space-wise going forward past the single machine level, meaning we can't just assume "and we'll add more machines if data builds up."
I'd have to test at what level (in our system) we'd hit a wall with the throughput for a single-node setup.
And when it comes time to scale out you still won't be able to afford the commercial version?
Or you could write your own sharding proxy or layer in your application.
Is the double-write architecture the nginx solution shown here? https://influxdata.com/high-availability/
That will be released and documented with 0.12, which should be in April.
What I meant by basic was that it would be a highly available cluster but not have other features that we have planned. It'll have recoverability, hinted handoff, anti-entropy for eventual consistency and a management and monitoring UI. It will most likely be limited by the number of cores.
Other features like role based access control and other applications we haven't started building yet will be in other pricing packages.
We haven't yet figured out the pricing for scale out clusters and those other features.
Even though this announcement is sold as something that will empower InfluxData to add even more cool features to the open source version, I'm actually worried that it won't be around as an open source product for very long. Let me explain.
InfluxDB is a very fine product but I don't think it has all the momentum it needs to keep going in the long term yet. That means it's still building that critical mass of supporters in an open source ecosystem. With this announcement, InfluxData is eroding the trust that small to medium-sized users had in it so the momentum slows down. If InfluxDB was a no-brainer for anyone starting a time-series project, it is not anymore. All the other alternatives need to be carefully analyzed and, even if InfluxDB is chosen, there is that voice in the back of your head saying you'll be in trouble if you exceed a single node's capacity. Eventually people will jump to the next TSDB solution that offers clustering as soon as it becomes avaiable. Then where are all those paying customers going? It'll then be easier and cheaper for InfluxDB to become a proprietary software company.
While there are some comparisons being made to what Nginx does with its Nginx Plus offering, I think it's the opposite situation. IIRC, since the early days, Nginx has offered a paid product with more features. And recently they have started to add those features to the open source version, so people got happier. InfluxDB has always promised clustering (it was a major selling point, go watch any presentation about it from 2 years ago), shipped the code (even if it's half-working as of now) and now announced it's removing the functionality. Suddenly any InfluxDB node you have (even if it's running just fine without clustering) looks like a lemon.
It's all such bad PR. And it's 2016, haven't hundreds of OSS companies been through this already? Oh well.
Single node usage is great for most people; sure it can be useful to cluster and you have that option.
If there's limitations it's open source; those can be removed. Then people would likely use the fork that doesn't have that limitation.
I don't see evil here. Just trying to work on their product
I run the tech side of a decent sized startup right now. Having joined ~4 years into the venture, one of my immediate concerns was the lack of visibility into how our system works. Keeping tabs on the performance of several hundred ETL pipelines is not super easy. I decided to double down on InfluxDB, changing the road-map of my infrastructure team.
Then, InfluxData introduced the TICK stack. I can get monitoring and alerting as well? Let's double down again!
I bought hardware, set up a cluster (painful), we filed bug reports, learned what sorts of queries not to run, and we were in good shape. I purchased tickets for a training session and a flight (out of pocket; we don't have travel budget). And then I saw the blog post last night.
All along, I knew that the promise of all of this great technology for free was too good to be true. There was always a little voice in my head asking me about how they actually made money -- I knew that a paid offering + professional services was not sustainable.
I'm angry, but we'll be fine. We're going to break apart our cluster and shard. We've already got code written to do a backup/restore that actually works (slowly) from one cluster to another. If we hit the point where sharding doesn't work, then it'll also probably be financially viable to pay the $399/month (on top of server costs). I'll still go to the training, but I'm not sure what I'll get out of it. The reason why I was going to the training was the section on "Cluster Administration".
The only thing that I'm raw about is the lack of apology. You have to make money, and you had to do a bait-and-switch as a result. I totally understand, but the tone of that blog post was too defensive and unapologetic.
Feel free to drop by our Slack (http://getkudu-slack.herokuapp.com ) if you have any questions.
InfluxDB 0.9 still had plenty of bugs, and I'd rather see a high-quality standalone server than any not-quite-there-yet clustered version.
The current version is siginficantly better than 0.9 and it's only getting better
This is part of the game called software. In some worlds this is actually the goal; that the software works so well and is so reliable that your customers eventually don't all need to keep paying you.
So you find new customers and add new, never advertised before features and enterprise clients with SLA support contracts, etc.
It's very understandable that it's hard to monetize, but giving people the impression clustering was going to be included long term and taking it away is not going to score you points.
Furthermore you want to get all the folks doing a startup on a budget hearing a story that works for them. Most ( especially new or young ) cofounder programmer types rarely plan to not need clustering, even if the reality is most won't need or use it. Saying this is for the big kids only is a turn off at this point in the influx story.
Wish you the best of luck
Are there many winners with this model to name. Maybe NGINX, MongoDB?
Whenever you see "open core," you should read it as "VCs said we need to do this to keep them happy."
VCs love the idea of how you can give part of something away for free then lock people in to auto-renewing subscription contracts for the whole banana (with fees increasing 7% to 17% every year for ongoing "maintenance"). Many VCs still actively recoil at the thought of giving away everything for free then just hoping the best will happen.
Nginx is a weird example because I'm not sure how well they are doing or who would buy "nginx pro" instead of haproxy+nginx or haproxy+varnish+nginx some other combination of already existing stuff.
MongoDB is an interesting example because they went with the "provide open software, but with really serious, game-stopping flaws, then force people to pay for support because nothing works" route. That's almost literally bait-and-switch (bait-and-support?). But, that hasn't ended up so well for them. From a recent WSJ article: Fidelity has cut its valuation of MongoDB in eight of the nine quarters since Fidelity made its investment in December 2013, valuing the shares 58% below what it paid.
Do you have a link to the MongoDB WSJ article? They are pretty terrible, even with a support contract bugs have a way of not being resolved or addressed.
I think this is a reasonable model. It's not easy to build / sustain complex software and while it works for a lot of projects, maybe majority of projects get neglected or depends on a few dedicated contributors. Who often burns out etc.
It also provides some confidence to users (at least to me). I know the project is open source. If the backing company goes evil, a fork can always emerge (iojs comes to mind). And I also know that some people are making money over this software and they will keep it alive / work on it as long as it sustains the business.
We also have a fully functional Grafana driver in case the customer prefers it over programmable visualization that we ship.
By programmable visualization I mean a way of building dashboards using toml-flavored configuration [1] and templating language, as opposed to manual design.
[0] https://axibase.com/products/axibase-time-series-database/ [1] https://apps.axibase.com/chartlab/2ef08f32
Most of the "community supported" projects that I have used are libraries. Since shifting from developer to the devops team, nearly everything that I rely on is "company led": saltstack, docker, grafana, influxdb, ELK, ngnix, sensu, rabbitmq, etc.
That being said, we are now re-evaluating the use of influxdb.
You want to pull the latest images an unlimited number of times ? Sure, it is just XXX$/month
Nothing will stop people to build open source containers of the same product, but those won't be "official" and there won't be any update guarantees...
We use a single server on Influx's cloud service for some internal metrics and it has been running without any interruptions for >6mo.
Rackspace Metrics (the product team behind blueflood) is currently doing many times more than 300 metrics per second. Our goal of this year is to reach to a new scale from where we are at, driven from the needs of big internal adoption this year. So blueflood project would be live and kicking.
Here is the most recent update on Rackspace Metrics product. All the functional work are through blueflood: http://bit.ly/rax-metrics-mvp
Also, there is a hidden gem in monitoring unknown for most of the startups. My other product, Rackspace Monitoring, is a SaaS product designed to assist Rackspace business instead of making big bucks. If you have a Rackspace cloud account, which requires minimum $50 service charge, you will be able to use Rackspace Monitoring for free. You can use it to monitor any other servers you host anywhere. This means that if your monitoring budget is already above $50 (all cost considered), you really should look into Rackspace Monitoring.
To learn about Rackspace Monitoring, please go here: https://www.rackspace.com/cloud/monitoring/
Please also feel free to reach out to me. We are located at the same location of Rackspace Startup Program in San Francisco. We talk to startups all the time and always enjoy the learning.