Unofficial Guide to Datomic Internals (2014)
tonsky.me
tonsky.me
Probably the most advanced database for triple stores these days is RDFox ( https://www.youtube.com/watch?v=-DnmuHtywFs ). While datomic uses datalog for querying, RDFox uses datalog for database reasoning, and sparql, a w3 standard for querying. As you add data to the database you can infer new facts. If you want immutability, simply add data in append mode only with a timestamp. But this idea you can add the business rules/logic to the database, and have it incrementally apply that logic as you add data is a recent advance by oxford AI research.
I used DataScript for a while to get familiar with graph database querying. I was fascinated how easy it is to construct queries that mine obscure relations between distantly related entities. I hope I get to use similar tech again.
> Datomic does not manage persistence itself, instead, it outsources storage problems to databases implemented by other people. Data can be kept, at your expense, in DynamoDB, Riak, Infinispan, Couchbase or SQL database.
These things can't both be true.
It really depends on the definition of “do more”.
From what I understand, Datomic’s model is far more flexible than many other databases, and has built in time travelling capability due to its accretion of immutable data.
It’s architecture does, in fact, allow you to choose the storage provider, and it’s considered an external concern.
Those are very compelling reasons to use it, but part of the trade off is write-scalability, and potentially, raw performance.
So maybe it’s “do more” within certain limitations (which is up to you to decide if those limitations are a deal breaker).
Here’s a great talk about Datomic’s architecture for more details: https://youtu.be/9TYfcyvSpEQ
That also seems debatable for the exact same reasons. I don't think this would convince anyone that needs convincing.
In most situations this is probably fine, but if you have data that changes frequently it seems like this could slow queries down compared to an EATV or AETV index.
It's also likely that the people who made Datomic are both smarter about this stuff than me and put more thought into it than I have, so I'd love to know what the reasoning behind the choice of index is.
(PS @dang it would be nice to have (2014) in the title)
By contrast, Crux [0] uses two dedicated temporal indexes: EVtTtC and EZC (Z-curve index) to make bitemporal as-of queries as fast as possible. These are distinct from the various triple indexes, which don't concern themselves with time at all. (Vt = valid time, Tt= transaction time, and C = the document hash for the version of an entity at a given coordinate)
[0] https://opencrux.com (I work on Crux :)
There you can only retrieve the top layer and don't have to scan all historic data, it's only in-memory though.
New data gets written to the log for durability and updates the in memory portion for queries. Periodically indexes are rebuilt, creating new segments for current, and shifting historical data out of current. This limits how much of the log must be replayed on recovery, and allows garbage collection of data that falls out of the retention window.
It's not that dissimilar to solutions used by traditional mvcc databases.
To determine whether a Datom is being rectracted or added there is a fifth element in the tuple [0].
There are many similarities to modelling temporal data in SQL [1]. But Datoms are simpler and more open as you can freely build relations between them (composable), similar to a graph-db.
EDIT: You can comment out the yellow background image in the style editor and it becomes something reasonable
Even more hilarious is switching to “dark mode”.
So you have deep technical debt with serious scaling issues and bugs everywhere(Datomic/Nubank) and a burnout company(Datomic/Cognitect) get together, makes sense.
Burnout because their "Datomic Cloud" product didn't worked out, it was just a horrible complex AWS cloudformation template that force you to click through tens of aws webpages. It was more complex to manage and to dev for than on-premise but you still had all the same issues and bugs.
Nubank got into Datomic not because of Clojure, but the other way around, they got into Clojure because of Datomic. If you watch their videos, the reason they picked Datomic was because they think it had "time travel", which is quite different from having "history" of transactions, use mostly for auditing and troubleshooting, not for real time travel queries.
In the end, I guess things did work out for Cognitect, and Hickey is now laughing all the way to the bank.
I have being following Datomic for a year because of a system I inherited.
I agree with you that the Datomic cloud stuff comes across as being frighteningly complex. I think they probably just need to work on the documentation, like making it more obvious what the differences and tradeoffs are between the deployment scenarios.
Did you inherit a Datomic system that was previously developed by a small team or a small company? Because inheriting a system that's hard to understand and change transcends languages and databases. It is the tie that binds us all as software developers.
Not being able to perform writes to your database is not scary enough? it's funny how they phrased that bug.
Also hit 4-5 more that are there in the change log, let serious but still pretty bad and frustrating.
This was a internal application, the DB was not being stressed, a 4KLoC readable Clojure codebase.
Don't get me wrong I really like Datomic and its features but the implementation still has a long way to go.
On your last point, I agree that it still has a way to go. It's good for some (many?) production use cases now, as Nubank's success demonstrates, and hopefully with Nubank's resources it'll start to live up more to its promise.
I just wish Cognitect would allow people to run public benchmarks of Datomic to make it easier to evaluate its tradeoffs.
From the Datomic EULA here: https://www.datomic.com/on-prem-eula.html
It’s common enough to have a name: “DeWitt clause”. It sounds like IBM is the only major commercial rdbms vendor to allow benchmarks?
Basically my impression is that DeWitt clauses are common enough to be well-known, but still in the distinct minority. That's just an impression though.
My biggest complaint is performance for certain use-cases. Say if you're trying to pull a lot of attributes on hundreds of thousands of datoms it's going to be rather slow (even though it's supposed to be in-memory already). But again for these kinds of use-cases I'd probably go with a completely different kind of a database either way.
The story around deletions/excisions isn't that great either. Honestly the whole log/history aspect of Datomic sounds nice but never really used it other than for reverting stupid mistakes.
The #1 thing I love is the freedom of querying you get with Datomic. You insert your data in a way that makes sense for your data, and querying is pretty much a completely separate concern. For the most part you don't need to structure your schema around the querying capabilities of your database which I love. Say back in the day I liked Mongo because you could just insert whatever you wanted [0] but eventually you'd hit problems where you couldn't easily query your data (maybe it has changed over the years, no idea).
And the syntax is just a pleasure to work with. I'd love a version of Datomic that kept the same interface but dropped some of the more esoteric features in favor of performance.
Also I noticed some of the people reporting issues used the cloud version. Never used that so can't speak to that. On-prem is free and has all the features. As long as you don't redistribute it there's no problem.
[0] Yes in datomic you do have to have a schema. But it's pretty much a simple global list of possible attributes. If you need to add something later or make a change it's pretty straightforward.
However, I indeed find Datomic Cloud version unnecessarily complex for most applications. Probably it is still a good corporate sales product for Cognitect.. Datomic On-premise version is much more friendly for small-medium-somewhat-larger use cases. Cloud version is also an AWS thing, so locks you in there, which is also not good.
Most highly performance-sensitive code in the Clojure ecosystem is a Clojure wrapper around a Java core.
But yes as I said elsewhere, it would be great if Cognitect allowed people to post benchmark results.
My company supports ClickHouse, but there are many use cases where it's simply not the right solution.
But you’re right C* and CH are designed to do different things. I just found the difference in general performance across everything (startup, schema changes, throughput, query performance, optimization opportunities) to be quite pronounced. One feels like a race car, the other not so much.
We are currently replacing it with PostgreSQL to improve performance and scalability.