Rich Hickey on Datomic, CAP and ACID

There's a bit of a barrier to entry for Datomic because it really doesn't look quite like anything that you're familiar with if you're just used to SQL and NoSQL. The basic starting point is one big many-to-many relation, what in SQL might be one big table of (db_id, subject_id, verb_id, object_id, timestamp) of these "facts" which are (subject verb object) tuples, and these queries can be limited by the transaction timestamp. I'd say more about it but honestly, because it's closed-source, I've had real trouble figuring out what exactly is going on in the backend; a lot of the organization is done with namespaced keywords so I am not sure how exactly the datatypes are organized, but you can query based on subject, verb, or object quite rapidly. As Rich Hickey says, one instrumental thing is that these tuples are only inserted -- they are never updated or deleted. This enables a "many readers, one updater" architecture without violating consistency when you're updating the data: nobody uses a timestamp greater than their recorded one until the model (in the MVC sense) tells that view that it's ready to update. There is a mechanism for naming new elements in your transactions so that you can insert a bunch of facts at once.

It's also a little difficult because it is substantially meta -- instead of a special syntax ("CREATE TABLE" etc.) for handling structure, the information about how the database is organized is stored in the same "fact" tuples that make up the rest of the database, just with some automatic verbs that come bundled in. I'm always a fan of self-expressing systems, but that might throw some newcomers for a loop.

I guess my main comment about Datomic is, "I wish there were an open-source version out there." I'm interested in seeing what the plumbing of such a system looks like and perhaps learning something from it -- and I'd like to peek at their Datalog engine and so forth. Unfortunately, just from peeking inside the JAR, it appears that one might need a good understanding of Google Guava and Apache Ant and the Jetty server and a bunch of other things to be able to read the source, which might make it prohibitive to peek inside the code. I'm more interested in learning from the system than I am in using it.

datomic claims to solve the O/R impedance mismatch -- the "vietnam of computer science"[0] -- which is why datomic matters to application developers.

"... The facts the Storage Service returns never change, so Peers do extensive caching ... Once a Peer's working set is cached, there is little or no network traffic for reads ..." [1, paragraph 4]

this basically means that your reads are VERY fast -- in-memory fast -- which means you can program as if your data wasn't in a database.

[0] http://www.codinghorror.com/blog/2006/06/object-relational-m... [1] http://docs.datomic.com/architecture.html

ps video is only 14 minutes long, worth a watch

From http://www.datomic.com/:

    Datomic is a database of flexible, time-based facts, supporting queries and
    joins, with elastic scalability, and ACID transactions. 

    Datomic can leverage highly-available, distributed storage services, and puts
    declarative power into the hands of application developers.

the more i see people speak about databases this way, the more i learn to appreciate chris date

almost all the explanations i've seen of datomic explain the software architecture or component (check this for example http://www.datomic.com/overview.html)

the more important aspect of a db is the conceptual data model, the abstraction it offer , the one which you as a developer will use to describe , create and query data and information

the relational model is not about the components of a DBDMS, its not about indexes, and optimizer, its about describing your data as relations, using relational operators and is even more primarily about data integrity

if it is not an implementation of the relational model, what is the model its trying to implement, and why should we use this data model?

I have immense respect for and look up to anybody who can do something as complex and low level as DB kernel development (mem management, persistence/file system, caching), especially with the long list of features you have to support to make the DB desirable/useful (ACID transactions, connectors).

I also hope DB implementers know how to get acquired by big, companies, because I don't see how they can really compete with these big legacy companies that have been developing their core DB's for decades and have an army of support/sales to back it up. Not to mention, where's the safety and accountability with mission critical data with a new product versus say DB2 or oracle? Maybe the new DB is a better implementation/more fault tolerant, but there's no big corporation to blame if something goes wrong, rather a small company you gambled on.

One thing I haven't heard discussed about the "save data forever" model is that in some scenarios you WANT to purge old data and not have it recoverable, for legal reasons.

I guess Datomic is not suited to these situations?

I think most people don't know that Datomic can be seen as a layer between your code/date and nearly any backend (including SQL: for example you can use PostgreSQL "behind" Datomic).

"CRA" (Create Read Append) is the future for 99% of all the application out there IMNSHO. I know most people don't believe it but you're simply not coming back once you understand how easy it is to "recreate the state" using such a tool. In addition to CRA, the fact that queries are forever cacheable is huge. The (lifelong) problem of "cache invalidation" simply doesn't exist anymore.

I'm not saying that Clojure is going to replace Java/C# or that Datomic is going to be used everywhere. But the concepts at the heart of these (like, say, lockless concurrency in Clojure and the "grow-only" property of Datomic) are what the future's gonna be made of.