Hacker News Reader

Comment by Confusion | Hacker News Reader

Poses the question

  So what’s the big deal? People have been recording 
  temporally oriented data since we could chisel on tablets.

Never answers it, but instead explains how Riak handles large time series. Certainly interesting, but I would like an answer to this question, as I don't understand the big deal.

View on HN

Immediately following the part you quote (my emphasis)

  Well, as it turns out, thanks to the software-eating-
  the-world thing and the Internet of Things we happen to be
  amassing *vast quantities* of all sorts of data [...] The
  demand for systems that are capable of storing and
  retrieving temporal data on an *ever increasing scale*
  necessitates systems that are specifically designed for
  this purpose.

Strongly implying the big deal is the need to scale?

Thanks, ya I thought I answered my question there ;)

That's an obstacle that needs to be overcome because you want to get somewhere. A challenging obstacle that has spawned an entire industry, but nevertheless not the goal. It is implied we can get somewhere new and exciting with these vastly larger timeseries. My question is: where?

I think Datomic which has been doing time series data for years now answers the "what’s the big deal" question for me:

"Given a value of the database, one can obtain another value of the database as-of, or since, a point in time in the past, or both (creating a windowed view of activity). These database values can be queried ordinarily, without having to make special queries parameterized by time." http://www.datomic.com/rationale.html

So in other words you get to see exactly how the whole database looked at any given moment through history (certain parallels with the blockchain I suppose).

Until you reach 10B datoms and then you're off for a fun ride sharding with multiple databases. Also in an IoT context or more general timeseries usage you very likely don't care about tx, but very much do about write throughput and not having a spof, which Datomic isn't good at at all.

Datomic is an ok choice in some contexts, but the one detailed here is not one of them, you're likely to reach Datomic limits quickly and be in a world of hurt when you do.

> I think Datomic which has been doing time series data for years now answers the "what’s the big deal" question for me:

This is known as "time travel queries". Postgres supported this back in the original version from the 1980s but then they took it out in 1997 because it takes up too much disk space.

It's trivial to do in an MVCC system. You just turn off garbage collection (vacuuming in PSQL parlance).

I think the big deal is allegedly "we now have so much data that we have a use for distributed databases with high performance", which allows them to go on and say "and our big deal is that we've built a high-performance distributed database tuned especially for time series".

I only noticed it because I was shocked that there was an actual comprehensible value proposition in a blogpost about a NoSQL product.

However being system admin of a PostgreSQL database managing a lot of timeseries I was just like "meh use timestamp and good index, what's the deal?". Also for performance sake just cluster the data using the most used index...

But yeah I guess if you really need NoSQL this should be nice. However sharding based on time will probably be one of the easiest approach for horizontal PostgreSQL deployment.

One of the most compelling features of Riak, both KV and TS, is that it is masterless and replicated. When nodes fail for whatever reason, the cluster handles it transparently.

Reply on news.ycombinator.com