From my limited experience with time series, not being able to insert data older than X seems like a big disadvantage that may bite later down the road. Bugs happen.
However for IoT or hardware time series I’m left with a few questions... A few things it seems are not possible:
- An IoT device with limited network connection that batches data points while offline
- manually loading time series data from disk at a later date on an offline device such as a weather station
- loading data “in the future” (data points with bad/erratic timestamps, memoized predictions)
I guess the hardware / IoT use case is pretty small compared to infrastructure monitoring, where Timestream’s design choices make a lot of sense.
OSIsoft was bought earlier this year by AVEVA for $5 billion[1]. OSIsoft's offering is centered around storing industrial control system data (the buzzword-version is IIoT, though that is a bit more broad). They're also not the only player in this space. Many of the industrial automation vendors also have their own offerings.
[1] https://www.bloomberg.com/news/articles/2020-08-25/aveva-to-...
Backfilling/inserting old data is common. Control network connectivity is not always great. Sometimes you need to do fill in old data.
Forecasts are also a common use case (e.g. forecasting grid demand/capacity), so not being able to handle future data is another limitation.
I'd heard that it was initially developed for a major industrial operator (sounded like it was in the resources industry though I never found out who it was for), so the lack of support for late arriving data is a big surprise. That being said, there aren't any of those companies on the customers page - maybe it was just an incorrect rumour.
[1] https://victoriametrics.github.io/vmagent.html#use-cases
You can't insert future data more than 15 minutes into the future though.
Memory-based storage is expensive, but not ruinously so.
The reason why this needs to be in-memory (I'm guessing) is a combination of dedupe checks and efficient ordered logs in the on-disk journals.
Additionally, you can scale the memory retention policy up and down as you see fit. So if you notice a network outage, you can scale up the retention policy for the duration and then reduce it back down again after.
Industrial facilities don't always expect their outages.
> Additionally, you can scale the memory retention policy up and down as you see fit. So if you notice a network outage, you can scale up the retention policy for the duration and then reduce it back down again after.
Better, though still a hassle to manage.
But we are working on making it easier. :-)
[0] https://docs.timescale.com/latest/using-timescaledb/compress...
[1] https://victoriametrics.github.io/#backfilling
[2] https://victoriametrics.github.io/#how-to-import-time-series...
[3] https://medium.com/@valyala/insert-benchmarks-with-inch-infl...