Thinking in Datomic | Hacker News Reader

Thinking in Datomic

pelle.github.com

96 pointspelle29 comments

The temporal aspect seems to be somewhat ignored by mainstream db development - in some fields, e.g. BI you'll have slowly changing dimensions; and the event sourcing pattern promises a time-machine view on data.

Since I'm writing a history-aware application at the moment, I recently looked into different patterns for this and trying a mixed strategy at the moment (SQL DB used as document store and a single event log, that accumulates changes - a lean approach though, a few hundred lines python for the data access layer; what always gets ugly is the validation, which your application must take care of).

I wish, there was more hands-on material on the subject (some resources dive depth into bi-temporal modeling, but I feel your schemas can get complex (= expensive) very fast).

by my understanding, datomic's persistence[1] layer is just a big huge persistent[2] data structure. there is tons of literature, books, papers, etc about persistent data structures in the functional programming community.

[1][2] 'persistent' means different things here.

So, Datomic is a persistent data structure persisted to nonvolatile storage... I've wanted one of these for at least the last 2 years.

If only there was an open source implementation of this concept that I could run on my own hardware. Does anyone know of such a beast?

And, no, I don't want to roll my own versioned/timestamped row schema in an RDBMS - been there, suffered that.

Acid-State for haskell (http://acid-state.seize.it/) is based on a similar ideology and in the public domain. (Oh, but minus the time dimension I'm afraid.)

Acid-State is quite different, it persists representations of operations to disk (to make it durable) and keeps the data structure in memory.

I'm working on a project to bring persistent data-structures to disk ( https://github.com/DanielWaterworth/Siege ). It's currently undergoing a rethink, hence the lack of recent activity, but it's still in development.

At small (one machine) scale you could store simply store entity$attribute$time as the key for the value in any sorted key-value store e.g. levelDB.

(that is, concatenate the entity, attribute name and timestamp in a lexically sorting string)

At larger scale, hyperdex http://hyperdex.org/ might do the job but I don't know much about it

key-values stores like levelDB are persistent [1], but they are not persistent [2].

[1] http://en.wikipedia.org/wiki/Persistence_(computer_science)

[2] http://en.wikipedia.org/wiki/Persistent_data_structure

Big Data by Nathan Marz is a Manning Early Acces book (i.e. not fully written yet) that explains how to hand-roll your own custom Datomic style DB in Hadoop/HBase.

It might be of interest to you given your current project.

Thanks for the tip, will take a look at it. Probably after this project, the current app must be lean (no big dependencies, just install a database scheme, pip install and go).

kdb+ is head and shoulders about the rest when it comes to introducing time / order into relational data. It's lean, minimalistic and ridiculously fast.

Unfortunately, it is also ridiculously expensive.

It's also easy to bring to its knees with seemingly innocent queries, and has a pretty darn opaque programmable interface in Q. (OK, the vaguely SQL-like stuff is not too hard, once you understand the differences from normal SQL, but doing anything lower level than that is awesome/crazy)

For what it is worth: http://en.wikipedia.org/wiki/Temporal_database

A more extensive description/case study of bi-temporal database design: http://www.cs.arizona.edu/people/rts/tdbbook.pdf

This case study tried to be somewhat rigorous, but from what I recall the SQL did get complex and expensive pretty quickly. There are a few attempts to create products to mask the complexity as well:

http://www.timeconsult.com/Software/Software.html

For living in time and dealing with it regularly in a common sense sort of way, it is really rather challenging to manage effectively as data.

There have been numerous triple stores. Mozilla/Firefox even used one as its core backend for a while before ripping it out and replacing it with SQLite. Why do you think those failed, and how is Datomic going to avoid the same failures?

That is, I've seen this pitched as the solution to all our data woes numerous times now, why is this time different?

It's not a triple store. It's closer to an OODBMS focused on persisting large object graphs with an immutable append-only datastructure, thus maintaining full history and being quite scalable with a single-writer many-reader configuration.

It is actually implemented more like a triple store than an oodbms. Just with time added as a 4th parameter.

Their Entity class makes it feel a bit like an oodbms though in that it kind of acts like a hash map.

Could you explain how it's more like a triple store with a time parameter?

My previous explanation already papered over a ton of fundamental differences in order to make a somewhat understandable generalization, but I can't make the jump to triple-store from my understanding of Datomic.

I basically see it as "just a single-writer many-reader distributed fully persistent data-structure."

I'm sort of sorry to turn this around on you, because it's generally poor form, but given the contents of the linked article which describes Datomic exclusively in the form of timestamped triples, can you explain how it's not a triple store?

If it's because you store objects as many triples with the same subject and different adjectives... err, well, yes, that's how you store objects in a triple store. Not remotely new, and still subject to my initial question.

The article leads people to confuse the chosen view (triples) with the model (versioned object graph.)

The main difference is that in Datomic the timestamps don't identify a single fact but the entire object graph at that point in time.

To make an analogy; saying that Datomic is just a triple store is about the same as saying that git is just a store for lists of diffs.

Triples are a simple data model that can be used for web-like graphs, relational tables, and hierarchical hashes. This is needed irrespective of Datomic's prospects. Just look at the the current massive adoption of linked data. http://news.ycombinator.com/item?id=3983179

Couldn't you just model your data in a traditional RDBMS with a time stamp as well? In fact for mutable data that you'd like to keep old versions of, this is pretty standard. A simple design would be to have a separate table for person_locations that mapped a person to a location.

With everything else, the standard RDBMS table could be considered as having a 'snapshot' of the Datomic values.

I'm still not sure what benefit this has over a traditional DB. Perhaps I'll just have to wait for the next post.

Of course you can make something similar if you're a) willing to build it and 2) don't care about the tremendous amount of complexity you're adding to your application in order to do so.

Both of those sound like bad deals if there's a packaged solution like Datomic that's built specifically for the use case.

You can do it all yourself if you want, but Datomic does it for you without you having to do anything.

Also as it does it on a datom level it is a lot more efficient than versioned rows.

Interestingly one of the early selling points of Postgresql was this time travel functionality. But it was yanked out in 6.2

http://www.postgresql.org/docs/6.3/interactive/c0503.htm

Yes, but you also loose the flexibility of not doing it. I'd rather work in an established RDMBS and configure in the design I'd like to use rather than use a database that requires a specific configuration.

I'm sure that Datomic has it's place, but the examples you gave in this post aren't that convincing.

I agree with you this is not on its own necessarily a good reason to switch to Datomic.

I will be getting into more detailed examples later. My real point with this post was more to talk about how to model the data using datoms and not specifically the temporal aspects of it.

I made many mistakes in my original data models in datomic based on many years of rdbms thinking.

The difference between Datomic and RDBMS isn't JUST a timestamp FYI. In fact I'd expect/hope this blog post is a just a prelude to Datomic's transactional model which is the real differentiator.

Absolutely. That is where I'm going in the next article. The transactions are extremely cool.

I look forward to it.

Yes, you can do similar in a traditional RDMS, you have a table called Person and another table called Person_properties with a schema of pid, property, value, timestamp. The object model for Person would have a save() and load() method that knows which properties belong to the base table (base properties) and which properties spill over in into the triple table. The separate properties table has the convenience of stashing arbitrary persistent properties in a Person without needing to migrate the underlying schema or change the base definition.

"Datomic is so different than regular databases that your average developer will probably chose to ignore it."

Playing to the Well I'm obviously above average gut feeling. Cute.

Reply on news.ycombinator.com