Migration from Postgres to Datomic
grishaev.me
grishaev.me
Datomic packages up a CQRS + event-sourced architecture in a really nice way. You can stash graph/tree/relational/tabular/(insert shape here) data and query it out with the powerful Datalog language. You get time travel (versioning) for free. Etc, etc.
It's a crime that this hasn't taken off as one of the leading data stores; hopefully Cognitec has some sort of roadmap to an open source release.
No string literals, pure data. Add transit to the mix and you get full-stack killer combo with unparalleled productivity.
With so many parts from Cognitect filling the stack it would be silly not to boost adoption with a friendly licensing.
Do you have an example?
So then where is the open source version / extension to Postgres?
How about this - I'll give you $1000 if you can give me a seamless extension on top of Postgres that has most of the functionality of Datomic.
Since it is trivial it is easy money for you.
Where is the evidence that you do not [insert anything here]? That's a slippery slope of challenge.
As a concrete example I like the way in the datomic schema you can eliminate bridge tables for many-many relationships using :db/cardinality :db.cardinality/many and then you store or get a vector.
There should be a way to have a super-SQL middleware compatible with any language bindings including a bare CLI that eats "SQL plus some cardinality" and behind the scenes implements the bridge table manipulations. Don't repeat yourself and all that, and many/many tables are kinda repetitive.
There doesn't seem to be anything remotely like that out there.
Fuck it, hyperfiddle isn't launched but lets just post it here. Hyperfiddle is JSFiddle + Datomic. http://hyperfiddle.net/ If you care about this, you should reach out, please email me!
For better or worse, history has proven that doesn't appeal to many developers. Lisps just aren't popular, and that lack of mindshare matters.
It's not helping clojure grow as much as it could though and it's a bit a chicken/egg problem, more clojure users could be more datomic users and vis-versa.
I would only make a closed-source solution a centerpiece of my business if:
* The thing is known to work for a number of other people, and uniquely solves my urgent problem, while no open solution can't do anything comparable. This is probably the Datomic niche.
* The thing was around for ages, and is a cash cow of a major and reliable software maker, and is also significantly better than open software in some important area. This is how MS or IBM sell their databases; Datomic doesn't have nearly enough mindshare to compete here.
* I'm hastily building an MVP that I plan to scrap anyway when the growth hits / the startup is acquired. Datomic is likely too expensive for that.
By necessity, the first case holds for a small number of businesses.
I've been considering trying to get up to speed with Mentat[0], which is a Mozilla project to create a Datomic-like layer over SQLite for desktop applications, and help out with that or even try to start a full open source clone of Datomic, though probably won't happen for a good while due to real life happening.
It's got so many good ideas and it sucks to see them go to waste in a proprietary database where basically nobody gets to use them.
What if it's unobviously not possible?
edit: found it https://soundcloud.com/defn-771544745/23-the-right-honourabl...
This was a common model before FOSS took off and it is still quite common in enterprise products.
If company X goes out of business you get the latest source version you paid for.
Having said this, I don't know if Cognitec has such kind of contracts.
Does anyone have any advice how to approach this? Example contracts, good experiences with escrow services maybe? Or bad experiences of course.
Datomic has perpetual free licenses so if they go bankrupt, you just keep the software you already have.
If you had an on-prem setup or a hosted white label solution, then it'd be easier to license.
I do feel that in a really well-run operation, your build, deploy, and operation should be so automated that that's just another set of repositories to put in escrow. One way to look at it is that escrow is just another kind of disaster recovery; if you have safe copies of everything you would need to restore your operations if you lost all your current servers, then it's easy to put those in escrow too.
It's not really effective for saving a cloud hosting. A platform like that is not easily redeployable.
True, but...
> This was a common model before FOSS took off and it is still quite common in enterprise products.
No, it wasn't and it's not. “Shared source” for paying customers was common before Free/Open Source Software (FOSS) took off (“Free” doesn't modify “Open Source”, they are essentially synonyms that identify the same license features but which are preferred by different ideological factions that prefer those features for different reasons.) Shared source does include access to the source, and it may include some rights to create derivatives (usually with no or restricted distribution rights), but it is not open source.
Also no one ever referred to it as "Shared Source" in the 80's.
Just to dig out one such product, here is the brochure of Turbo Pascal 5.5 from 1989.
https://archive.org/stream/bitsavers_borlandtur5.5Brochure19...
" Turbo Pascal 5.5 Runtime Library Source
Modify the runtime library source code or use it as it is. You get the assembly language and Pascal source code to the System, Dos, Crt, Printer and Turbo3 units. It comes with a batch file to help with recompiling and building TURBO.TPL."
For me, for my work, that was open enough.
It's no religion; the terms have well-established meanings.
> What matters is having access to the code, whatever form it might be.
The legal permissions you have to make use of the code very much matter, too. And those vary considerably among non-Open Source, source available/shared source license arrangements.
It's essentially a black box now, you have no auditing possibility, no way to extend it, no guarantee you can pick up the pieces is cognitect/rich goes away. You basically sign a check, and hope for the better.
Note: I too wish Datomic was OSS, just disagree that price isn't the issue. It is.
We tend to gloss over the fact that when dealing with OSS:
- Project maintainers can leave/change/stop supporting the software and the idea that "since it's OSS, I can just pick up the slack" sounds better in ones head, then it does in practice.
- Even large projects (think Angular) can move in odd and unexpected directions, which in reality simulate a company "going out of business" for all intents and purposes.
- OSS 'support' is great, until it's not. Meaning, if there are plenty of SO questions and answers, it's great, until your specific issue isn't addressed, at which point you're either looking to a third party to pay for support, or you're rolling your sleeves up and digging into a codebase you have no experience with outside of an external API level understanding.
The dirty secret of large successful companies is that they actually pay for software, and the support of that software because they understand that time = money and vice versa.
After using datomic in production, I'd welcome my competitor to use Postgres. I'd even push them to. It's a monstrous competitive advantage for me.
There is an "open" strategy, when you offer the core part of your cool thing widely (cheaply / freemium model / for free), and profit from only e.g. 10% of the users that buy premium plans. You also have a fair chance to make your better way of doing things predominant, and put in the hands of a lot of people that won't pay you either way.
The point is that when your cool thing may have a really wide adoption, with the "open" model you may have e.g. 50x the market share of the "narrow niche" model, and thus 5x the profit. Increased mindshare also helps further adoption. This, of course, depends on the adoption rate and paying customer rate.
Haven't you noticed the popularity contest going on with computing?
If datomic isn't popular it will disappear.
The more important question is how many key patents are involved. That is, how feasible a comparable open-source project might be.
create table snapshot (
id uuid primary key default uuid_generate_v4(),
ts timestamp without time zone default now(),
entity_id uuid references entity(id),
value jsonb
);
create table entity (
id uuid primary key default uuid_generate_v4(),
snapshot_id uuid references snapshot(id)
);
I did a proof-of-concept of this years ago in this linked comment (which is the bottom of a long thread which involved Rich Hickey; I remember it being an interesting thread).
https://news.ycombinator.com/item?id=4448767
At the time I did not provide a patch (which is weird for me) to show how "this is totally just leveraging stuff already there", but I just went and found that folder, so here is the patch:
To expand: when you have to query a remote database for data, you likely only want to perform one query for performance reasons (and certainly not an unbounded amount of queries - maybe two or three is acceptable but not N+1). This means that information must be passed down the call stack for what information is needed in that single or few queries, creating coupling between unrelated layers of your application.
To make this more concrete: imagine you write a function to find users named Jim. At first this function is for reporting, so you just return a list of user IDs. Later, you decide to build a dashboard. You want to render all users named Jim here, but you need each of their names for display purposes.
Given a remote database, you now need to modify the query function to be able to return the specific attributes you need for this use case. You can imagine if you extend the call stack this passing gets more complicated, requires merging of the queried items, etc.
With datomic, since your data is in memory, your original query can stay the same, since N+1 queries are irrelevant when your data is available in RAM.
I think this point is important.
Also, I may not be getting what Datomic is, but why modify original function, you can just join results with users table.
In that scenario you could join the user table. You'd be overfetching in many scenarios but that's not a huge concern for most people, this is what active record does. However, say you want to get more than just the columns on the user table, then you run into the same issue. Suddenly the query caller needs to inform the query method to include results about some unrelated table. Because datomic is an in memory graph structure, the caller can handle grabbing that extra information without modifying the method or its call signature, obviating the need for this coupling.
It all depend on your need, RDMS schema are customely build to meet customers needs, I don't believe in proverbial Swiss knife tool anymore.
1. When you have multiple entity tables that are inter-related, snapshotting one in isolation is not sufficient.
2. When only certain changes on certain fields should result in a new snapshot.
Granted, both of these are solvable with built-in Postgres mechanisms (and a bit of co-operation from the application server) but simple it is not.
In our case we also move historic records to a second 'history' table.
I'm curious wether Datomic supports RE and how.
The features I like most are
1) the transaction log - I've done something like that many times using Postgres and EventStore, but nothing beats the simplicity of just defining a few queries in code and having immediate updates on new transactions delivered to every peer.
2) idempotency - reasserting the same facts is a noop. Doing the same with a temporal table as suggested somewhere in this thread is not as trivial.
3) consistent db snapshots - once you get a hold of the database value, all the reads will only see the data as of time the value was retrieved. This makes application code much easier to reason about as the database can be treated as yet another immutable argument.
4) assembling a transaction value out of multiple pieces - same as the above, pure functions can all contribute to the final transaction value without having to mutate anything.
5) "free" caching on the peer - once you query something, it stays in the peer's memory. Subject to memory constraints, of course.
You can definitely build something as nice as Datomic on top of Postgres, but it will take weeks to get all the details right.
The database is fine; and it was nice to work with when it was just 2 engineers. For us, it didn't scale with # of engineers, business needs like BI, ETLs of large amount of healthcare data into Datomic; then needing to delete said data. We spent a lot of time reinventing the wheel around schema management and building a declarative query interface as well.
> few queries > take weeks
Godspeed.
To clarify - I wasn't trying to address the issues you've stumbled into, but rather list all the points I like about it.
> Avoid nils
>
> Datomic does not support nil values for attributes. When you do not have a value for an attribute, you should either skip it or pass an empty value: a zero, an empty string, etc. That’s why the most of expressions have (or "") at the end of threading macro.
You can also use `(missing? ?entity :attribute)` on queries, which should do a faster lookup on the EAVT index vs. checking for sentinel values.
> JSON data
>
> In Datomic, there is no JSON type for attributes. I’m not sure I made a proper decision, but I just put those JSON data into a text attribute. Sure, where is no a way to access separate fields in a datalog query or apply roles to them. But at least I can restore the data then selecting a single entity:
---
Datomic is a beast different enough to require some learning on what is optimal/idiomatic in terms of data modelling, because the paradigm shifts from appending/mutating tuples over multiple tables to asserting/retracting facts on what is essentially "one big table" (entity, attribute, value, transaction).
They do share some commonalities such as:
- history/transaction subscriptions and queries
- Providing a Django ORM/Model like data access system via Datomic's 'Entity API' (although this is Django)
But Datomic isn't just 'event sourcing' (although it's a way to think about solving a problem).
It provides Prolog style queries, functions on the database, a GraphQL like API, Caching (Memcache) and robust Peer library (which also makes it hard for NON JVM languages to use Datomic as it uses a closed protocol... (although very likely uses Artimes/Hornet MQ's internal protocol))
I am talking from a Clojure/Datomic background (so my Django experience is old & limited) But I'm sure you could cobble together something to satisfy all these things for your stack, Datomic just does a beautiful job for Clojure (And I find Cognitect's strangle hold on the technology not great but I think they've loosened the licencing enough to make it acceptable for small businesses & startups which strikes a good balance).