Hacker News Reader

Comment by cupcakestand | Hacker News Reader

OT and hijacking the thread:

"TimescaleDB (the OPs product) is a new open source time-series database built up from PostgreSQL."

Do you know good alternatives or which distributed databases are generally well suited for huge volumes of time-series data? Cassandra?

View on HN

Most products in this space are pretty obtuse, so I'm excited about TimescaleDB because of the people who started it and the tech it's built on. I was really underwhelmed by problems with Influx, and I don't think this group and this tech will have the same class of issues.

I think HBase is better than Cassandra for this workload, and I generally like its design sensibilities and consistency focus (gross oversimplification C* = MySQL, HBase = pgsql?) . OpenTSDB is one of the more mature scale out timeseries DBs and works with both. It works pretty well, but getting a production quality HBase setup is non-trivial. OpenTSDB has some decent built in aggregation and filtering functions, but queries over long time spans are quite slow because it has to pull all the data out of HBase and into the tsd daemon.. that can be fixed with what HBase calls a co-processor (think stored procedure in SQL) but it hasn't been done yet.

So, I think TimescaleDB is hitting a real need at the right time, and seeing how expressive and easy to reason those queries are, I am excited to give it a shot.

Obviously I'm biased, but in our testing of TimescaleDB we've found it to do well with large amounts of data. We're still working on benchmarks that we hope to present in the coming weeks/months, but we've been able to sustain high insert rates even when the database contains billions of rows and metrics. And in our preliminary comparisons to Cassandra the query latency for TimescaleDB was much better.

Happy to try and address any concerns you may have

Billions total? Per month? Per day? Per hour? Per minute?

I hate the word "large" in these contexts.

Our internal benchmarks (which we are publishing soon), show a sustainable insert performance of 100K+ rows/second (where each row contains 10 metrics -- some would call this 1M metrics/second), even at 10 Billion rows. All on a single commodity instance (with only 16GB RAM). (We will publish these soon, along with scripts to reproduce them.)

But I agree that "large" often means different things to different people.

You can check https://clickhouse.yandex/

How big does this data end up being? Couldn't you store it in a bunch of double arrays in memory?

Reply on news.ycombinator.com