Hacker News Reader

Comment by Toshio | Hacker News Reader

Supposing Datomic was open-source. Can you give a short summary of what you find so appealing about its design?

Honest question, I'm curious.

Many enterprise databases have a requirement where during the regular progress of business, database rows are never deleted or updated - instead, for every change, a new row with a timestamp is added with a copy of the old data, and then mutated. In other words, many, many business applications built on top of databases have half-assed version control of the data built into them.

Datomic takes the knowledge that many databases are built like this, and basically designs with that in mind. The ability to say "give me a view of the data as it was 1 month ago" is built into the system from the very beginning, rather than added on later.

There's also design pattern in systems based on relational databases that skips the need for copying so much data to keep history - instead of keeping primary data in your database with additional column "timestamp", you keep changes to that data in another table, and either use triggers triggered by inserting these changes to modify the primary data, or forgo primary data alltogether, and just use view over changes table (can be slow).

I've seen it implemented 2 times in enterprise systems and it works OK, but like all design patterns it's workaround for lacking feature.

Also some dbs (for example Oracle) allows you to do

    SELECT * FROM SOME_TABLE WHERE USER_ID = 5 AS OF TIMESTAMP...,

but that's not good enough.

While looking on the subject, I do recall something about old PostgreSQL versions having something like it... But I've always wondered how they deal with changes to database schemas, types, etc.

While googling I've found from this post: http://www.postgresql.org/message-id/Pine.GSO.4.64.080203152...

this free (?) book - Snodgrass/Jensen "Developing Time-Oriented Database Applications in SQL" - http://www.cs.arizona.edu/people/rts/tdbbook.pdf

Haven't read it, so I'll spend some time through it.

To me this is an interresting thing, since I would like to know whether I can do history tracking of game asssets for game development. E.g. often in our studio someone would like to compare data, how it was before this and what changed. We do have the mechanism, but they are not so ellegant.

Oracle only keeps history for specified time, it takes up space, on our systems it's usualy a few hours, so it won't work for long-period history.

Regarding game assets - why not keep them in git or some other cvs? For textual data it has huge additional advantage - you can blame, merge and easily see differences, but even for binary data it can be used.

To expand on Yuri's comment: the standard view of databases, with mutable data, is flawed in the same way that mutable data structures are flawed in programming. Mutable state makes reasoning about program behavior more difficult than necessary. Datomic applies the same concept to databases — modifying a record no longer clobbers the previously-recorded information. This is a great safety feature, aside from its applicability to any auditing or reporting requirements.

Mutable data structures are a much more accurate abstraction when it comes to how computers actually store data in memory and on disc -- I don't think you can reasonably call them "flawed." A (properly-implemented) mutable memory structure will have better performance characteristics by definition.

Don't get me wrong, I like immutable data and I use it over mutable whenever possible (Clojure is my favorite language), but to call it "flawed" is an overreach at best.

Mutable data structures are a very accurate abstraction of how computers work. They are a very inaccurate abstraction of how the world works. At any given instant, the world is constant; time moves forward, and changing the world as it was 1 second ago is not possible. Insofar as most of business programming is trying to efficiently give answers about the state of the world at various points in time, I would agree with gcv that "flawed" is the appropriate term to use for mutable data structures.

> They are a very inaccurate abstraction of how the world works.

No, they are only inaccurate software abstractions of the business abstractions they are supposed to model. People like immutability and reversibility of concepts and that's why they try to apply it in business, law, finance etc.

But the real/natural world, at least from our subjective perspective is not like that. Things in the world are not practically reversible or immutable - you cannot "restore" a deadbody. The real world is loosy and mutable, at least from an agent's perspective. The human brain is so efficient at what it does simply because it just "throws way* (irreversibly!) so much information every second, filtering for just what it needs. A small corner of the world is our "business/finance world" and using immutability and reversibility is great here, but there's way more to computing than implementing business/finances logic and processes in software!.

...if you think about it, classical OOP is inspired from cellular biology (can't find the reference now, some musing of Alan Kay I think...), so it's no wonder why it's such a trouble maker when used to model business logic, it was the wrong tool for the job from the get go. But generalizing that mutable data structures are a bad tool to model and understand the world in general is waaaay overstretched, the world is bigger than "business", and all modeling approaches are equally useful.

Unless I'm missing the point of your argument, it sounds like you're saying that, e.g., computational biology does not benefit from representation as immutable data structures. Although I have no experience writing software for modeling cell or biological entity behavior, I strongly suspect that reasoning about the simulation itself by representing its states through immutable data structures, or immutable databases, will lend itself to a better piece of simulation software.

If you think I'm wrong about this, I'd appreciate some examples.

Mutability is change without chronology, which isn't the way the real world works. Immutability over time is a more accurate abstraction of physical processes, particularly when you get into smaller time increments and services distributed over larger distances.

Mutability is useful as an optimisation, but I don't think it has much worth as a model.

>The real world is loosy and mutable, at least from an agent's perspective.

No, it's not. You cannot mutate any fact! You will never be able to change the fact that Hitler existed, for example. Facts never change.

This is the purpose of Datomic: to store facts. Mutable databases only seem to be interested in maintaining the most recent fact about any particular piece of information.

Your comment is not a response to the comment you replied to. In particular, it was quite clear about the flaw in question:

> Mutable state makes reasoning about program behavior more difficult than necessary.

You could dispute that claim, but bringing up performance characteristics is totally irrelevant to it.

Mutable data structures are a much more accurate abstraction when it comes to how computers actually store data in memory and on disc

Yet programming in imperative, mutating, mainstream languages is nothing like how code works at the machine level. I understand you're talking about data, but I think the analogy still stands. Or maybe a better one is that a file system or even a database itself is an abstraction of how data is stored on disc.

Besides what has already been said, I am interested because it is one of few distributed, consistent databases capable of transactions. Google Spanner, Hyperdex and Postgres-XC are the others I know of; they appear to be quite a rare breed.

There are other distributed consistent databases (HBase, RethinkDB, arguably Cassandra if you always use QUORUM, etc.) but neither supports transactions or has a design that seems to me like it easily could.

Is it distributed nowadays? I vaguely remember Datomic as having a single-point-of-failure "transactor" which every request had to go through. (Given the specialized name, maybe the transactor itself doesn't have to do very much; but it still makes me wonder how you'd distribute the data store over many nodes if the nodes aren't capable of transactions themselves.)

Writes have to go through the transactor. Reads don't. So Datomic is a bit of a hybrid; writes are single point, reads are distributed.

The storage layer is distributed automatically because it's decoupled from both the transactor and the peer. It's actually pluggable, too... you can use DynamoDB, Riak, Couchbase, Infinispan, Postgres, etc. which all have different performance and availability characteristics.

I'm ok with serialised writes for most purposes.

I see the created_at and updated_at fields as the poor man's datomic.

Reply on news.ycombinator.com