Hacker News Reader

Comment by kev009 | Hacker News Reader

Be _very_ skeptic of this, I wasted weeks trying to get git master versions of this not fall over including tsm1 storage engine. They are very good at marketing and made nice query and API entry, but data storage has been a total shit show.

Looking at less than a month of issues:

* https://github.com/influxdata/influxdb/issues/5440

* https://github.com/influxdata/influxdb/issues/5482

* https://github.com/influxdata/influxdb/issues/5534

* https://github.com/influxdata/influxdb/issues/5540

If you need something that won't fall over, it's not glamorous and a bear to setup, but OpenTSDB will sail with massive load once you've gotten it running.

View on HN

Not sure what issues you're seeing, but we have people running this in production at significant scale.

One of the issues you linked to was for 0.9.6. Others were there because they had super high tag cardinality and not enough memory to actually run.

When people post comments like "I have 50 million series and it crashes on my box with 2GB of memory!!!", they're not relevant. You shouldn't expect miracles...

How about issues like these? https://github.com/influxdata/influxdb/issues?utf8=&q=is%3Ao...

In my case I had a 0.9.4 influxdb setup fed by statsd for about a week. The server hard crashed the first time I tried to back it up.

We just have to close those out. Those are all fixed in the release we have today.

No, and you're countering rhetoric with rhetoric so here are the specs: bare metal 96G RAM, 16 cores, 36 disks (also tried 36 SSDs). Nothing exotic, and something similar you should be using to test before releasing another ill-fated storage engine based on "It works on my Macbook Pro with an artificial load test, ship it!". On the same box carbon is usable, and TSDB can scale linearly across a cluster of these machines. Influx burns down as soon as more than a few GB of metrics are persisted even after backing off collection.

I'm not quite sure what you're doing. If you have that level of hardware I don't know why you're running out of memory. Our tests with 100B points split across 1M series ran on much more modest hardware than that and didn't have a problem.

There's something different with what you're doing than with what we're testing. If you can give more detail about what your actual data looks like, I may be able to help.

Are you doing this on v0.10 (beta1 or greater) and you're seeing this problem?

I don't have a lot of time to spare on this and we've found a viable option so I won't be using influx any time soon. I can help you with access to my production system if you are interested and as you are watching arrange for between 1k and 10k hosts (for 2 - 20 values per second)

I'm sad that pauldix disappears when the comments get direct like this - I've used influxdb at smaller scales and I'm genuinely curious to see responses to a few different threads going on in this discussion. But they are left without a reply as of now.

I disappeared because I can't spend all evening responding to every person on the internet that may be having trouble with our software. An offer to SSH into a system that isn't running Influx and trying to troubleshoot some ambiguous problem isn't direct.

We have people that have given us useful troubleshooting information that we've helped (and have helped us improve InfluxDB). Constructive criticism with an offer to help us improve is the best.

While I'd love to trace down every problem for everyone on the internet, my time on this planet is limited.

As for your concerns about if it scales, we'll try to put out more benchmarks and test code over the coming weeks. For reference, we built a tool for stress testing that you can see here:

https://github.com/influxdata/influxdb/tree/master/cmd/influ...

You'll have to make your own decisions.

We would install Influx, as a go app that is fairly trivial and it speaks the same collection protocol...

As a random problem on the internet, this is the workload you should expect from any midsize business and probably something your company would like to target for revenue. Good luck.

This is why I said they are great at marketing. There's really little value in working with me, but at least responding while the article is front page makes it look like they are responsive and looking into issues.

Maybe they can sink a big user/customer that's willing to gut through all the pain of helping to make an operation data store.. someone in this thread mentioned mongo and it sounds very familiar.

I'm wondering if I'm being too negative but the flip side of this is that it caused me and others a lot of stress, lost face, and time. Most developers and operators are overworked already so I am trying to save some pain since the blog post makes it seem like everything is fine and awesome.

He replied an hour ago...

An hour after I said that, yeah. But not in the like 6 hours prior while he was answering other stuff.

This is a very odd thread.

I'm in no way involved in influx, we're just evaluating it, but this entire thread reads like complete FUD at the moment, so it's odd that you're calling out a maintainer or whoever pauldix is for "disappearing".

If you're just now evaluating it, you might not realize the history behind the project. It's more backlash than FUD.. infrastructure people don't like to be jerked around by marketing tactics any more than we already are. At various times each of the storage engines was promoted as the bee's knees LevelDB -> BoltDB -> bz1 -> tsm1. That kind of fumbling speaks for itself. It's strange for a high profile investment (including YC). Founded/funded around the same time contrast historically to FoundationDB.

> Founded/funded around the same time contrast historically to FoundationDB.

FoundationDB was in stealth mode for at least three years before they launched their closed Alpha release, so perhaps this isn't a fair comparison.

(I was an intern at FoundationDB during summer 2012.)

At the time I wrote the comment, there were direct questions left unanswered for 6 hours, while replies were given to other less-interesting comments in the meantime - that to me looks like someone ignoring something. He came back, that's great!

If you found something else that works for you, that's great. I'm sorry that InfluxDB didn't work for you when you tried it, but we'll continue to improve and build. My feeling is that this release is a significant milestone in our evolution. We'll see how people react over the coming weeks after actually using it.

> If you have that level of hardware I don't know why you're running out of memory.

Did you folks fix the "oops, I accidentally selected too much and OOMed the machine" crash that was supposed to be resolved by the new query planner in .10? That was my immediate showstopper for InfluxDB for a very high-volume infrastructure, because I didn't feel like proxying InfluxDB just to enforce chunking, and the mere existence of the bug gave me pause. I hit that in 24 hours and shelved the system in 48.

Even pre-1.0, I'm not encouraged when I have to "work around" pretty obvious oversights in reliability, so I put you back on the back burner to give you time to mature and transitioned back to a Kappa-style architecture for my needs. I might still use InfluxDB for low-volume aggregations out of my stream processing, but it's for sure out of the hot path for the foreseeable future.

That is more or less how my employer worked around this glaring bug. That being said, we've worked together with Influx to fix some issues or add features to Telegraf and even have had their developers fix several of our issues found under excessive load (/me waves to Alex and Sebastian), Influx has a long way to go, but I've got faith that they're going in the right direction and things are getting a lot better. It is true this announcement is a bit too full of blatant marketing rhetoric, but the tech is good and if is getting better. If you've not tried it very recently, give it another shot.

The current version still has the problem that if you do a massive SELECT then it will fill up memory until the process gets killed. In the future we'll give controls to limit how much memory a query can take.

I think it's common in databases that if you throw a massive query at it that the server doesn't have the resources to handle, things will go wrong. It'll thrash, or crash, or generally have poor performance. We'll be working on improving the failure conditions, but if you put a query to a database that is too big for the server to handle, some sort of failure scenario will be encountered. Just a question of how it's handled.

Also, there's no new query planner in 0.10. We have a bunch of work getting merged in for the query engine for 0.11. But if you throw a huge query at the DB that the server can't handle, you'll still have great sadness.

Which DB ended up solving your big query problems? Maybe we're just a poor fit? Or what kinds of queries should we be working on optimizing?

> I think it's common in databases that if you throw a massive query at it that the server doesn't have the resources to handle, things will go wrong.

This has never ever happened to me in the last 20 years I've used SQL databases. Sure it might take a long time but that's it.

The current version still has the problem that if you do a massive SELECT then it will fill up memory until the process gets killed.

That means any ad-hoc query can trivially crash your server. That's pretty serious. It's even worse if the ingest is using statsd-influxdb-backend with udp since during the crash you'll lose data.

Do you really think it's ok for a novice user to be able to crash the server from your web gui if they make a minor mistake (e.g something like

  SELECT value FROM /series.*/

not realizing how many series their regex actually applies to until it's too late and the has gui stopped responding)?

I think it's common in databases that if you throw a massive query at it that the server doesn't have the resources to handle, things will go wrong.

This thinking is worse then wrong. Not only is it wrong - a simple unqualified select won't crash any "common" database - it makes anyone with any "common" database experience doubt your other claims.

> Did you folks fix the "oops, I accidentally selected too much and OOMed the machine" crash that was supposed to be resolved by the new query planner in .10?

> The current version still has the problem that if you do a massive SELECT then it will fill up memory until the process gets killed. In the future we'll give controls to limit how much memory a query can take.

...

It locks on the query until its killed manually and/or kills it automatically.

It doesn't crash.

Reply on news.ycombinator.com