Sorry, but at that point, we've decided to rebuild the entire metric visualization once on TimescaleDB, since we're running postgres a lot anyhow.
Solutions like Grafana Mimir, Victoria Metrics, Clickhouse, or yes, the new Influx implementation, are much more scalable and will give you much fewer headaches.
ClickhouseDB is realy brilliant, btw, it's a powerhouse. Especially with the fairly recent additions that enable hybrid local + S3 option, pushing older metrics to S3 for cheap long-term storage.
It’s fantastic for workloads that neatly fit in the hypertable pattern though.
https://news.ycombinator.com/newsguidelines.html
Are you expecting a real answer of how I was hoping timescale's internal watermark system would help me roll up a total count or are you just implying I'm an idiot?
My issue was that for a grand total I didn’t have a time column, so I couldn’t define my query as a continuous aggregate and the query had to start counting from the start of my underlying series each time.
mike (at) timescale or DM on twitter?
But there are lots of approaches, depending on your needs.
You can (should) define a "cache disk" for S3, which will cache up to X Gb locally to avoid trashing.
Another option is is to move data into separate (purely S3 backed) tables after a certain time to avoid accidentally fetching large amounts of data from S3. You can still easily join the data together if needed.
https://www.timescale.com/blog/expanding-the-boundaries-of-p...
If not, I would suggest looking at a proper OLAP DB. VictoriaMetrics has been great and was easy to set up.
Are managed "proper OLAP DB" solutions competitive with managed RDBMS from a price and ease of use standpoint?
This link has a comparison of features[1].
[1] https://docs.timescale.com/about/latest/timescaledb-editions...
And beyond that, TimescaleDB works with a few things we have already. We could migrate Zabbix to use TimescaleDB for a large performance boost. Also 1-2 teams are building reporting solutions for the product platform, and they are generating some significant timeseries data in a Postgres database as well.
My bad experience with TimescaleDB 3 years ago was that enabling compression required disabling the "dynamic labels" feature, which was a total nonstarter for us. A proper timeseries DB is designed to achieve great compression while also allowing flexibility of series. Hopefully Timescale will/has fixed that without adding another drastic perf tradeoff, but given how Postgres is architected for OLTP I would be surprised.
Just solve compression on the block level, why are you so specific about it happening in the database? It’s probably one of the least interesting feature comparisons when betting on which database to trust.
What is the "dynamic labels" feature? Is it a part of Postgres or Timescale?
I assume it's doing an automatic ALTER TABLE when necessary, which modifies each row and somehow breaks compression across the sharded tables. Or at least an automatic re-compression would cause massive latency on insert that they wanted to avoid.
The most common approach here is just to store the step of "dynamic" labels in JSON, which can be evolved arbitrarily.
And we've found that this type of data actually compresses quite well in practice.
Also regarding compression, Timescale supports transparent mutability on compressed data, so you can directly INSERT/UPDATE/UPSERT/DELETE into compressed data. Under the covers, it's doing smart optimizations to manage how it asynchronously maps individual mutations into segment level operations to decompress/recompress.
(Timescale cofounder)