A Fast Lightweight Time-Series Store for IoT Data

I was a little disappointed to see that this was just a single-machine hierarchical time index on top of a pre-sorted append log.

It's not a bad architecture, and the lockless queuing system and shared memory is cute, but:

- It only supports time indexing. Querying by other fields (e.g. if you wanted to build a time histogram of when you saw a particular event) requires reading the entire dataset.

- It doesn't address replication/distributed indexing, which seems like a must.

- The use of calendar time over micro-timestamps also needlessly complicates things, though presumably it makes their queries more efficient (assuming people query by discreet time buckets like "yesterday").

with the risk of sounding cynical here, but it feels this is a solution from people that aren't familiar with what the problem is about, ending up in optimizing the wrong areas.

Few observations here:

* using udp means you don't care if you lose data and that's bad. if your buffers are full your data will be silently dropped, aside from network glitches and metric sizes that can easily exceed 512 bytes. Your buffers can get full when you wait on disk io etc..

Optimizing your network stack when your bottleneck is disk io doesn't buy you much. But the real issue that they haven't tackled is data distribution. If you can horizontally scale your data optimally, then when you hit a bottleneck (whatever it is) you can always increase your cluster size and solve your bottlenecks. In the current solution, If the machine dies, you will lose tons of data.

cow pipelining exist for a very long time now. There's nothing new in what they are introducing.

The persistent data they hold in memory is not guaranteed to be recovered if they crash before an fsync/msync.

Secondary index on time column is not enough. Queries often need particular keys only and you'll end up doing full table scans.

I'm surprised Cassandra isn't mentioned. It's not specifically made for time series and there are some tricks to make it work nicely (bucketing) but it's a solid solution.

Incidentally I was reading an old HN post today on Kdb+ for timeseries where people who use that commercial db said it's amazingly fast (I only played around with it, nothing speedy). People in that discussion mentioned that 300k writes/s mentioned about Kdb+ is nothing compared to the millions of writes/s IoT solutions need. So I am wondering what kind of performance you can get on time series databases on a single machine; I understand that with clustering systems like Cassandra can do 100 million/s and more (someone mentioned billions but not what DB), but getting benchmarks just throwing hardware at the problem, to me, only makes sense if we know the performance on a single node, multicore performance single node and then the scaling graph when adding nodes.

I am just interested in this kind of thing; I used to work with time series for a German telco; that was DB2 on heavy IBM metal for massive amounts of money and financial work for a startup. I wonder what the state of the art here is.

Correct me, if I am wrong. InfluxDb is written in Golang and not Java as the table 1 mentions in the paper.

In the industrial control system world we call time series DBs 'Enterprise Historians' they are well optimised. Good ones support storing data at native frequency of control system including sub-second intervals. Basically whatever frequency the instrument can sample at these things can record.

They also have good compression and buffering support, including catchup functionality.

They are by no means lightweight - usually they rely on PLC's or SCADA system for inputs. They may not offer a complete solution for IOT but I'm sure there are lessons which can be adapted.

Does anybody know why the paper states that influxdb is implemented in Java?

Prometheus InfluxDB opentsdb ... and on and on

It seems it's pretty crowded

The primary dataset used for testing is "USGS archived earthquake data". Does anyone have a link to the actual dataset?

These need the traditional YA prefix. Like YACC, YATSD or YATSS?

druid??