Hacker News Reader

Comment by theomega | Hacker News Reader

I can only recommend TimescaleDB. It solves the right problems (storing timeseries) while not creating new ones (deployment, backup, hot failover) as it relies on Postgres to provide the underlying infrastructure. I stored 100 million sensor samples in TimeScale and had not issues with scaling on medium sized boxes, despite issuing complex time-series queries.

As for the hosting option, currently sadly AWS doesn’t offer Timescale as part of RDS. There are two options: Azure offers Timescale now as part of their hosted Postgres. Or you go with aiven.io who can host you postgres with TimeScaleDB on all cloud providers (AWS, GCP, Azure, DO, ?) as a service, including replicas and backups.

Overall, I’m very happy to see the Postgres ecosystem growing.

View on HN

If you are looking for hosted TimescaleDB, eg on AWS, you may want to check out Timescale Cloud [1], which we launched a couple months ago.

Fully-managed TimescaleDB, including community and enterprise capabilities, high-availability, etc, available on AWS, GCP, and Azure.

There are other options as well (Azure, DigitalOcean, Alibaba, Aiven) but they only offer the OSS version of TimescaleDB.

More here (including a pricing calculator): https://www.timescale.com/cloud

[1] https://blog.timescale.com/blog/timescale-cloud-first-fully-...

Thanks for the recommendation! And glad to hear TimescaleDB meets your needs. I wanted to add some color to hosting / deployment options around TimescaleDB.

Options are as follows:

[1] Fully-Managed TimescaleDB Enterprise on Timescale Cloud

TimescaleDB Open Source hosted on Public Clouds:

[2] Azure PostgreSQL

[3] DO Managed-Postgres (mentioned above)

[4] Alibaba Cloud

Not hosted, but managed service options:

[5] ClusterControl from our friends at Severalnines

---

[1]https://www.timescale.com/cloud [2]https://azure.microsoft.com/en-us/blog/power-iot-and-time-se... [3]https://www.digitalocean.com/docs/databases/postgresql/resou... [4]https://www.alibabacloud.com/blog/sql-and-timescaledb_595169 [5]https://severalnines.com/blog/advanced-database-monitoring-m...

Interesting. My team currently uses (abuses?) postgres for timeseries data. You mind ansswering some general questions about your experience with timescale? You said 100 million sensor samples. What was the upload/download frequency? Our application is pushing hundreds of millions of rows across many different data sources every day. On top of that, we are also querying the shit out of this data to run models and we need VERY quick queries. like 10-100ms speed.

How do you think timescaleDB would handle that size and also velocity of data?

actually the last time I checked/used it timescaledb did use parallel queries and had a good partinoning so that the parallel queries would even scale really good on a single node.

TimescaleDB could fit your workload if PostgreSQL fits you. The main issue with PostgreSQL and TimescaleDB is big amounts of storage space required for huge time series data volumes. There are reports that storing data on ZSF can reduce the required storage space.

Probably, ClickHouse [1] would fit better your needs. It can write millions of rows per second [2]. It can scan billions of rows per second on a single node and it scales to multiple nodes.

Also I'd recommend taking a look at other open-source TSDBs with cluster support:

- M3DB [3]

- Cortex [4]

- VictoriaMetrics [5]

These TSDBs speak PromQL instead of SQL. PromQL is specially optimized query language for typical time series queries [6].

[1] https://clickhouse.yandex

[2] https://blog.cloudflare.com/http-analytics-for-6m-requests-p...

[3] https://www.m3db.io/

[4] https://github.com/cortexproject/cortex

[5] https://github.com/VictoriaMetrics/VictoriaMetrics/

[6] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...

We are actively working on native compression in TimescaleDB, with really promising early results. More to come.

DigitalOcean Managed Databases also comes with TimescaleDB built-in: https://www.digitalocean.com/docs/databases/postgresql/resou...

I was going to post a very similar comment :)

For on-prem deployments I'm using TimescaleDB on a single node with up to 100 million events, and for SaaS I'm using it on Azure, and the performance is little short of amazing!

Something I particularly like is the almost instantaneous deletes (because of the "chunking" model) - perfect for data retention jobs that delete old data.

I use RiakTS although unfortunately with Basho's demise it's future is in a bit of jeopardy.

Reply on news.ycombinator.com