Comment by koito17 | Hacker News Reader

Comment by koito17 | Hacker News Reader

Parent Full thread

Professional Clojure dev for some time. Here is what prevents me from using Datomic.

- no Docker image; still distributed as a tarball. Although com.datomic/local exists, it only provides the Client API, so it's mostly suited towards mocking Datomic Cloud.

- Datomic Cloud is designed around AWS; applications wanting to use Datomic Cloud must also design themselves around AWS

- Datomic On-Prem does not scale the same way Datomic Cloud does (e.g. every database has an in-memory index; the transactor and *all peers* need to keep them all in-memory)

- No query planner whatsoever. In databases like XTDB, the order of :where clauses doesn't matter since the engine is able to optimize them. In Datomic, swapping two :where clauses can transform a 10ms query into a 10s query.

In addition to the above four points, I strongly believe the following points prevent others from using Datomic.

- Writing Datomic queries outside of Clojure (e.g. Java) requires writing everything in strings, which feels awful no matter what.

- If you are not using a JVM-based language, then there is no choice but the REST API for interaction. The REST API is orders of magnitude slower than the official Clojure and Java clients.

- Too many tooling exists around the pgwire protocol and various dialects of SQL. Datomic obviously does not fit into either of these categories of tooling.

- Applications like DBeaver do not support Datomic at all. The best you can do is access the datomic_kvs table from a database using JDBC storage engine.

> In databases like XTDB, the order of :where clauses doesn't matter since the engine is able to optimize them

Rich's observation was that query optimizers often get things wrong in ways that are hard to predict or control, but he's not fundamentally opposed to their use. That said, building a decent optimizer is a huge undertaking and I think they took the right decision to not attempt to bundle that sort of complexity into the original vision for Datomic otherwise they might never have shipped.

The state-of-the-art commercial engine for query optimization and execution algorithms in this 'triple' space is probably https://relational.ai/

As I see it, Rich's stance is that of an expert in the database that doesn't need to deliver business features using the database. New users are not experts and even experienced users that work for companies have pressure to deliver features. You can get initial popularity by targeting these types of expert users working on more experimental products. However long term growth and popularity requires targeting the other 99.9% of users. I've seen one company adopt Datomic due to this type of user and then a couple years latter rip it out because as it grew it's developers were no longer of this type.

The original RDBMS vision was very explicitly for the users (both developers and analysts) to not have to be experts in their own database in order to achieve useful work, and without needing to think about procedural/3GL code from the get-go. In the intervening years query optimization has gotten a lot better, and hardware shifts have only worked in favour of this vision, but there's still a lot of work to be done before databases are truly "self-driving": https://www.cs.cmu.edu/~pavlo/blog/2018/04/what-is-a-self-dr...

Until that's the case I can understand why people are tempted to bypass this traditional RDBMS wisdom, especially if they have a very strong conception about their data models, access patterns, and need for scale (e.g. see also Red Planet Labs 'Rama').

If you don't have RDBMS wisdom and program an application around a RDBMS, your application is going to suck.

No ORM or framework is going to save you.

In my experience, structuring a query to execute efficiency requires some basic software-engineering thinking about what's going on, while convincing a query planner to do the right thing requires deep expertise in the query planner.

> convincing a query planner to do the right thing requires deep expertise in the query planner

The advantage is that that can be somebody else's job though, and ideally (eventually) an AI's job.

I kind of agree, I find it was never that easy to set up and use. I also found the documentation to be quite limited on specific points I wanted to know.

They showed of a JOOQ like Java API for Clojure once but as far as I can see, this was never released. That is crazy to me, using it amazingly well from Java and friends would seem to me to be an absolute no brainer. That alone made it basically impossible to be adopted. Going from SQL/JOOQ to Strings was just not gone happen.

They focused so much on Datomic Cloud, and that just isn't where most people are gone deploy. Specially in the age of Kubernetes and Docker. Its kind of crazy that there were not official Docker images and things like that.

So even while I love Datomic conceptually, and once you have it set up with Clojure its pretty awesome. I would hesitate to really use it for a larger project.

I would really love if NuBank simply open-sourced it.

As someone without Clojure experience but eyeing Datomic from the sidelines, thank you for the detailed answer, super interesting!

For an example of Datomic in containers, https://github.com/carrete/datopro-mic

Would you reach for XTDB instead of Datomic?

>No query planner whatsoever. In databases like XTDB, the order of :where clauses doesn't matter since the engine is able to optimize them. In Datomic, swapping two :where clauses can transform a 10ms query into a 10s query.

Datomic has query-stats https://docs.datomic.com/reference/query-stats.html

Reply on news.ycombinator.com