Datomic Free Edition
blog.datomic.com
blog.datomic.com
It is a database system for Java or Clojure where not all of the database code runs on the server. Instead, the server is fast and "dumb" and all the work for your queries runs on each client individually, which just cache chunks of data from the servers as needed. It turns out having your queries run on the clients has nice properties in terms of query language power and performance.
Additionally, it has an extremely powerful and elegant way of tracking historical information: No data is ever forgotten, so that if you want to say "run this query against the database as it was last Wednesday" you can do that. In a typical production application, much of the complexity in the database comes from having special auditing tables that keep track of historical info- In datomic you don't need to have such tables.
Finally, it can do all of these things while still being highly scalable (with a small caveat that it might not fully scale in the rare case where you have an app that is extremely write-heavy.)
I'd also add a sentence about transactions being first class. Everything is wrapped in transactions, you can add metadata to transactions (who did it, when did it happen), and since no data is lost, you can use transactions IDs to see what your database looked like 1 week ago, etc.
Also I like the idea that queries can invoke your own code. (Which is feasible since the queries execute on the same machine as your app.)
This may sound like heresy, but in practice the following line is going to be a showstopper for most people:
Peer.q("[:find ?entity :where [?entity :db/doc \"hello world\"]]", db);
Take this lesson from cassandra. They started out with Thrift and an extremely cumbersome query-interface, which fit the cassandra data-model perfectly but not the brains of the developers.
Now recently they exposed their query-language as a SQL dialect, and suddenly cassandra is a joy to use.
You should do the same. It will be a lossy abstraction. You'll need weird, non-standard constructs to accommodate the peculiarities of your model. Purists will cry in horror.
BUT: It will look like SQL and roughly work like SQL. Everybody and their dog already knows SQL. People can jump in and bang out "select foo from bar ..." without thinking. People can re-use an entire galaxy of SQL-related tooling and knowledge that has evolved over decades. Most importantly: People can start with something they know and then adapt at their own pace to the great new things that your datastore enables.
Don't underestimate this if you're aiming for the mainstream.
Also, they are big on developing systems that are very modular, and since datomic is built on Lisp, adding a SQL frontend would certainly be very easy (like a weekend project.)
I'm sure once these guys have higher priority features addressed, you'll see a SQL feature as well (if folks from the clojure community don't beat them to it and create it first.)
There are already enough SQL databases in the world. rhickey isn't looking to make a 'me too' database, he's interested in improving the state of software development.
SQL is a terrible language and needs to die. It's fine that we disagree, but don't tell the people who are convinced it's a bad idea that they need to 'join' the mainstream.
SQL is a terrible language
No disagreement here. Just out of curiosity, what would you call this language then: [:find ?entity :where [?entity :db/doc \"hello world\"]]?
What exactly do you see as the problem with this example? Or with SQL for that matter?
Honestly, I'm interested in what your criticism is. Is it the syntax?
Obviously I'm not entitled to tell anyone how to design their databases. I'm just saying there's good reasons why so many DBs stick to a SQL-like syntax, and that is because the alternatives are usually worse (think: familiarity, tooling, scripting, REPL, etc.).
For example, using the MongoDB REPL, which is probably close to what a datalog-REPL would look like, is rather painful.
This may all be a non-issue when a LISP-language is exclusively used on the client-side. That query-style probably just snaps in naturally there (I don't know lisp).
But if the database is supposed to be general purpose, accessed with lesser languages, scripted, quickly fixed by half-drunk humans at 4am in a REPL... then it certainly matters to have a sane intermediate lingo. Otherwise, at the least, every client-platform is going to invent their own.
I think the example of Cassandra is really a good one to study. They went through an interesting learning-process that seems very applicable here.
Edit: Please see my other comment above. It seems I have misunderstood the project goals and this is in fact not meant to become a general purpose DB. Under that premise my concerns obviously don't apply...
SQL is what people are familiar with, and even some NoSQL databases have distinctly SQL-inspired query languages (e.g. SimpleDB). The problem is that SQL is a godawful query language, and if we want to do better we need to do something different, and therefore unfamiliar.
SQL tries to look like natural language, which has resulted in a syntax is complex, inconsistent and monolithic, just like the language it tries to imitate. If we want a syntax that is simple, consistent and modular we need to throw away the idea that a query needs to read like English.
"Familiarity" is a terrible metric for new things. New things are not always familiar. As rhickey said in one of his talks, there's a difference between what's simple, and what's familiar. "I don't know German, does that make it unreadable?". German may or may not be the ideal thing, but the fact that it isn't english is not an argument against it.
> Just out of curiosity, what would you call this language then: [:find ?entity :where [?entity :db/doc \"hello world\"]]?
Datalog (or a dialect of it). Datalog is a variation on prolog, designed for querying databases. Just as there are many lisps, there are many datalogs. This one is interesting because datomic querys are valid clojure data structures. It's a great example of code as data. It's trivial to write clojure functions that return data that can be passed as a query to datomic. If you wanted, you could probably even write datomic queries that return other queries. Try doing that with SQL ;-)
I made my first comments under the assumption that this aims to be a general purpose database, but multiple people have made clear now that this is not the case.
Obviously it makes no sense to argue for an intermediate QL (and one as half-baked as SQL) when the project is ultimately aimed at Lisp-purists[1].
[1] This is not meant derogatory, it's just a critical distinction from a DB that, say, my junior-admin who knows his SQL and Python and not much else could be expected to get along with.
I recall reading about the ability to clear out old transactions, to account for fast-growing databases. For example, "delete all transactions and facts older than 2 months". 2 minutes of googling didn't yield any results though, sorry.
>> (2) can we modify values as-at past points in time? Example, an electricity company records usage monthly for customers, but sometimes past usage is incorrect and needs to be edited. What would be the best way to structure this to get the "updated" view as-of six months ago? And also the "original" view as-of six months ago?
If I understand you correctly, you do this by adding a new fact with the correct usage. Your queries will now return the latest number, and you can use `as-of` and so on to get the old value.
I am using Datomic on a friend's project right now. We will certainly use the Pro version, but a free edition will help promote the platform.
http://thinkrelevance.com/blog/2012/04/26/thinkrelevance-the...
The key to me is that you can run multiple queries against the same point in time, thereby having a sound basis for the data you are working with. Compare this to working with traditional RDBMS having to get it all done in a single query to have a coherent view of the world, or roll the dice by running multiple queries knowing that they all might not be running against the same view of the world.
b)It doesn't require any coordination, read locks, MVCC etc
I think the subscription model is a much better approach in todays market. In particular since I think Datomic's early adopters are going to be mainly startups and not larger enterprise customers.
Their initial hosted transactor model was a good idea as well. The problem I think was that it was hard to come up with a way of pricing it in a way that people could understand.
Here's hoping they will return to our add back either the subscription model or hosted transactor in the near future.
If I could just go to datomic.com, log in and click "create new database" and call it "my_new_app_database" and then immediately use that database from my app that would be fantastic.
(In short, I wish there was a heroku for datomic... admittedly, paying customers probably aren't impeded by such newb issues.)
I needed to copy link ( http://www.datomic.com/pricing.html ) & paste it.
The compatible APIs and relatively straightforward upgrade process also makes it appealing. Well played.
Standard web application using free edition. Three servers. Architecture is DB storing data locally, a web server and another server (say for slow report queries, an API or web worker). Could use beefy servers to scale up.
Business application using pro license allows better redundancy and scalability. Can use different data storage options. Can break beyond limits of 2 servers for processing and request handling.
The pro's peer license cost per process (which I think means cost per grunty server cost) is at worst $800 up front plus $400 per year.
Note: This is unlikely to be quite right.
Updated: thanks to RichHickey.
Also: you can backup from one storage system and restore to another. So if you decide you want to upgrade from local disk to DynamoDB, no problem.
They have their own query language but it looks to me that they did a great job at making it as close to SPARQL as possible, which personally I am familiar with.
Pretty neat stuff.
SPARQL is cool, and they did a good job making it look a little like datalog.
So, I clicked the link at the end of the page to go to your website. I followed some more links. I found three links that talked about feature, benefits, and the architecture, and clicked all three opening them in tabs.
I read the first part of every page and then skimmed the rest.
I have no damn clue what this is. Its something database. It has immutable facts from the past. Why would I use it?
All of your pages talk about features that might be important to someone who's considering the software, and about how it works, and stuff like that.
But never, in clear terms, what it is.
I have the same problem working on what we're doing here, what it is is new, so there's not an obvious "We're an X" or "We're an X with a Y" that I can give.
The best bet if that's the case for you, probably would be to do something like "Say you've got a Q and you need to W it while you're Z is already at X, if that's the case then Datonic will do P, O, and R for you giving you result Z!"
Or whatever is appropriate for whatever it is datomic is.
Just a constructive suggestion. Who would use it and why?
Edit: In response to those too pressed for time to watch the videos, you can read the rationale[2] for Datomic. However, it is a bit long. Rich and Stu may want to summarize it for the really pressed for time. :)
In my opinion, if a product can't be described in a couple of concise sentences, it's going to run into some major roadblocks in getting traction.
Watching a video incurs a number of costs - the time taken to watch the video, the effort of plugging in headphones, the pain of buffering content. All of these are minor obstacles, but you want to eliminate as many reasons to leave as possible when attracting a new customer.
It still requires watching a video, but the video "Simple Made Easy" by Rich Hickey (http://www.infoq.com/presentations/Simple-Made-Easy/) describes it best. If that doesn't appeal to you, then don't worry about what you're missing in Datomic.
The attitude that Clojure is only for some enlightened few who are worthy enough to understand it is extremely alienating. Lisps aren't that hard, homoiconicity isn't that opaque, the benefits of using Clojure can be explained in practical terms that most developers can understand, if not at first be convinced by. All these concepts can be explained succinctly in text.
Saying that Clojure is only for those who share some 'vision' is hand-wavy at best, insulting at worst.
Edit: My guess is that Datomic is targeted towards Clojure devs because they are already experienced with Datomic's philosophy and will understand the system better. It also provides a smaller and highly receptive market. They can then focus on perfecting the software instead of training a large number of people in the philosophy behind Clojure and Datomic.
I think we're in violent agreement.
Actually it seems like it's something that this hypothetical guy would be very interested in, given that Datomic is being pitched as an alternative to SQL.
I realize I'm a bit on the fringe, here, and I've complained about it before. But I do not like "basic" web pages, e.g. this blog post, becoming dependent upon local execution of arbitrary code (in which camp I'm not including HTML and CSS).
For me, it's a matter of security. And I'll continue to whitelist, based upon established trust.
Go ahead HN Lisp lovers, kill my karma!
http://www.infoq.com/presentations/The-Design-of-Datomic
I would probably have used it for the system I'm making now, as it's a very good match to our requirements, but 0.1 is 0.1, and the docs are kind of sparse so the learning curve for Datalog queries was too steep.
The transaction identifier column represents time which enables you to query the state of the database as of a point in time (e.g. yesterday, last month, etc.) for reporting. The subject-relation-object structure supports adhoc NoSQL-style changes to the schema. Transactions themselves are first-class concepts, so you can add relation-object pairs to them as well.
Their hypothesis seems to be that querying client-side data is faster than server-side data (obvious), and that updating data can be effectively synchronized by a dedicated transaction coordinator (potentially). Although, this is a single point of failure, the claim is that this code can be audited and perfected.