Hacker News Reader

Comment by kasey_junk | Hacker News Reader

Still waiting for someone to open source an observability db that uses distributions as the core storage abstraction instead of just single values.

I thought Circonus’ IronDB was open source but they’ve either brought it back closed source or I was mistaken.

In any case it seems weird that everyone is just rehashing the same space with metrics dbs and not moving to where Google/Circonus have been for a while.

View on HN

clickhouse as aggregate functions (Pearson correlation coefficient,quantiles , etc... )[0].

Associated with their Materialized view and Liveview features, you can achieve observability [1].

[0]https://clickhouse-docs.readthedocs.io/en/latest

[1]https://www.altinity.com/blog/2019/11/13/making-data-come-to...

Edit: fixed broken link.

That second link seems to be broken. The base URL is: https://www.altinity.com/blog/2019/11/13/making-data-come-to...

fixed. Thanks.

I'm not sure I understand. Google, Circonus, etc. all store distributions as scalar values. A histogram is merely a collection of scalar values, where the collection is a set of bounded buckets, where the boundaries are typically adjacent to one another (e.g., 0.1 <= x < 0.2, 0.2 <= x < 0.3, etc.). You can do this with any timeseries DB.

The aggregation functionality is best done in a separate layer so that you have the flexibility to change how you create your distributions without having to change your storage backend.

They store scalar values in that they are binary encoded data structures. But the data structures they actually encode at the cell levels they histogram bins in a datastructure optimized for histogram calculations (in the Circonus case log linear histograms encoded as 3 values[0]). These histogram structures are optimized for fast re-aggregation and quantization across time/dimension. By optimizing for histograms first and plugging things like counts and gauges into them (rather than the reverse for something like Prometheus) they get around a lot of the bad statistics and expensive query calculations that other metrics systems have.

This is different than say how Prometheus stores histograms where it treats the bucket boundaries as a dimension in a fixed bucket histogram and the cell as the count of items in that bucket.

[0] https://www.circonus.com/2018/05/effective-management-of-hig...

Thanks for the link! This is some good reading.

Veneur[1] can compute approximate global percentiles (among other things) during metric collection, and the percentiles can be stored and queried in even in a datastore that doesn't know anything about distributions.

[1] https://github.com/stripe/veneur

But not across dimensionality and because veneur has outsourced the storage of the metrics you can’t calculate arbitrary percentiles after the fact.

I've been working on something like this for some years now: https://github.com/zeebo/rothko

Reply on news.ycombinator.com