Hacker News Reader

Comment by qaq | Hacker News Reader

Prometheus InfluxDB opentsdb ... and on and on

It seems it's pretty crowded

Just submitted this: https://news.ycombinator.com/item?id=11654402

> It turns out you can accomplish quite a lot with 4,709 lines of Go code! How about a full time-series database implementation, robust enough to be run in production for a year where it stored 2.1 trillion data points, and supporting 119M queries per second (53M inserts per second) in a four-node cluster? Statistical queries over the data complete in 100-250 ms while summarizing up to 4 billion points. It’s pretty space-efficient too, with a 2.9x compression ratio. At the heart of these impressive results, is a data structure supporting a novel abstraction for time-series data: a time partitioning, copy-on-write, version-annotated, k-ary tree.

It's not at all crowded since not all of them actually work well and they are optimized for different use cases and different setups. One use case I believe is not covered well, and I use it often, and evaluated it, is actually storing the complete time series and reading it as a whole, without any aggregation. This is important for different signal processing and so on, where sampling does not cut it. At the and I chosen KairosDB (with Cassandra backend). Not that it is perfect, but it if CRUD is most important, and you offload processing to other engines, it works well.

That is why I am always looking forward to new things such as this or BTrDB. There is still a lot of room for improvement everywhere.

It's crowded because there isn't a clear leader.

Speaking as a Prometheus developer, each is suited to different things.

For example if you want to store timeseries data long term and already use HBase then OpenTSDB is a good choice.

On the other hand if you want to do monitoring that's simple and dependable in an emergency with querying, graphing and alerting over short/medium-term data, then Prometheus would a good choice.

> short/medium-term data

This in particular, but your entire post, would be a great addition to the Prometheus front page, or top of the documentation section. This wasn't clear to me initially, speaking as someone who evaluated prometheus a few months ago.

We plan on seamlessly integrating with long-term storage (https://prometheus.io/docs/introduction/roadmap/#long-term-s...) and OpenTSDB is one option for that for us.

Ignoring that as it's planned work, the typical considerations are more around availability vs. consistency and that we're a metrics system focused on operations rather than a event store. Most of that's already covered in our docs. See https://prometheus.io/docs/introduction/overview/#when-does-... and https://prometheus.io/docs/introduction/faq/

I wouldn't include opentsdb under "lightweight" as it requires HBase.

Last time I checked opentsdb only supported millisecond resolution timestamps. That's not enough precision for scientific data. I'm using influxdb because of the nanosecond timestamps, though microseconds would be enough.

Reply on news.ycombinator.com