Hacker News Reader

Comment by mtrn | Hacker News Reader

The temporal aspect seems to be somewhat ignored by mainstream db development - in some fields, e.g. BI you'll have slowly changing dimensions; and the event sourcing pattern promises a time-machine view on data.

Since I'm writing a history-aware application at the moment, I recently looked into different patterns for this and trying a mixed strategy at the moment (SQL DB used as document store and a single event log, that accumulates changes - a lean approach though, a few hundred lines python for the data access layer; what always gets ugly is the validation, which your application must take care of).

I wish, there was more hands-on material on the subject (some resources dive depth into bi-temporal modeling, but I feel your schemas can get complex (= expensive) very fast).

View on HN

Big Data by Nathan Marz is a Manning Early Acces book (i.e. not fully written yet) that explains how to hand-roll your own custom Datomic style DB in Hadoop/HBase.

It might be of interest to you given your current project.

Thanks for the tip, will take a look at it. Probably after this project, the current app must be lean (no big dependencies, just install a database scheme, pip install and go).

by my understanding, datomic's persistence[1] layer is just a big huge persistent[2] data structure. there is tons of literature, books, papers, etc about persistent data structures in the functional programming community.

[1][2] 'persistent' means different things here.

So, Datomic is a persistent data structure persisted to nonvolatile storage... I've wanted one of these for at least the last 2 years.

If only there was an open source implementation of this concept that I could run on my own hardware. Does anyone know of such a beast?

And, no, I don't want to roll my own versioned/timestamped row schema in an RDBMS - been there, suffered that.

At small (one machine) scale you could store simply store entity$attribute$time as the key for the value in any sorted key-value store e.g. levelDB.

(that is, concatenate the entity, attribute name and timestamp in a lexically sorting string)

At larger scale, hyperdex http://hyperdex.org/ might do the job but I don't know much about it

key-values stores like levelDB are persistent [1], but they are not persistent [2].

[1] http://en.wikipedia.org/wiki/Persistence_(computer_science)

[2] http://en.wikipedia.org/wiki/Persistent_data_structure

Acid-State for haskell (http://acid-state.seize.it/) is based on a similar ideology and in the public domain. (Oh, but minus the time dimension I'm afraid.)

Acid-State is quite different, it persists representations of operations to disk (to make it durable) and keeps the data structure in memory.

I'm working on a project to bring persistent data-structures to disk ( https://github.com/DanielWaterworth/Siege ). It's currently undergoing a rethink, hence the lack of recent activity, but it's still in development.

For what it is worth: http://en.wikipedia.org/wiki/Temporal_database

A more extensive description/case study of bi-temporal database design: http://www.cs.arizona.edu/people/rts/tdbbook.pdf

This case study tried to be somewhat rigorous, but from what I recall the SQL did get complex and expensive pretty quickly. There are a few attempts to create products to mask the complexity as well:

http://www.timeconsult.com/Software/Software.html

For living in time and dealing with it regularly in a common sense sort of way, it is really rather challenging to manage effectively as data.

kdb+ is head and shoulders about the rest when it comes to introducing time / order into relational data. It's lean, minimalistic and ridiculously fast.

Unfortunately, it is also ridiculously expensive.

It's also easy to bring to its knees with seemingly innocent queries, and has a pretty darn opaque programmable interface in Q. (OK, the vaguely SQL-like stuff is not too hard, once you understand the differences from normal SQL, but doing anything lower level than that is awesome/crazy)

Reply on news.ycombinator.com