Excision in Datomic | Hacker News Reader

Excision in Datomic

blog.datomic.com

69 pointsfogus39 comments

This is (was?) one of the reasons I don't use Datomic.

The other is that it's closed-source, which sadly has not changed. I certainly wouldn't mind paying for commercial usage/support. I can't justify relying on something I have no control over whatsoever when there are so many open source alternatives, however clean and nice I find Datomic's design to be :(

Supposing Datomic was open-source. Can you give a short summary of what you find so appealing about its design?

Honest question, I'm curious.

Many enterprise databases have a requirement where during the regular progress of business, database rows are never deleted or updated - instead, for every change, a new row with a timestamp is added with a copy of the old data, and then mutated. In other words, many, many business applications built on top of databases have half-assed version control of the data built into them.

Datomic takes the knowledge that many databases are built like this, and basically designs with that in mind. The ability to say "give me a view of the data as it was 1 month ago" is built into the system from the very beginning, rather than added on later.

There's also design pattern in systems based on relational databases that skips the need for copying so much data to keep history - instead of keeping primary data in your database with additional column "timestamp", you keep changes to that data in another table, and either use triggers triggered by inserting these changes to modify the primary data, or forgo primary data alltogether, and just use view over changes table (can be slow).

I've seen it implemented 2 times in enterprise systems and it works OK, but like all design patterns it's workaround for lacking feature.

Also some dbs (for example Oracle) allows you to do

    SELECT * FROM SOME_TABLE WHERE USER_ID = 5 AS OF TIMESTAMP...,

but that's not good enough.

While looking on the subject, I do recall something about old PostgreSQL versions having something like it... But I've always wondered how they deal with changes to database schemas, types, etc.

While googling I've found from this post: http://www.postgresql.org/message-id/Pine.GSO.4.64.080203152...

this free (?) book - Snodgrass/Jensen "Developing Time-Oriented Database Applications in SQL" - http://www.cs.arizona.edu/people/rts/tdbbook.pdf

Haven't read it, so I'll spend some time through it.

To me this is an interresting thing, since I would like to know whether I can do history tracking of game asssets for game development. E.g. often in our studio someone would like to compare data, how it was before this and what changed. We do have the mechanism, but they are not so ellegant.

Oracle only keeps history for specified time, it takes up space, on our systems it's usualy a few hours, so it won't work for long-period history.

Regarding game assets - why not keep them in git or some other cvs? For textual data it has huge additional advantage - you can blame, merge and easily see differences, but even for binary data it can be used.

To expand on Yuri's comment: the standard view of databases, with mutable data, is flawed in the same way that mutable data structures are flawed in programming. Mutable state makes reasoning about program behavior more difficult than necessary. Datomic applies the same concept to databases — modifying a record no longer clobbers the previously-recorded information. This is a great safety feature, aside from its applicability to any auditing or reporting requirements.

Mutable data structures are a much more accurate abstraction when it comes to how computers actually store data in memory and on disc -- I don't think you can reasonably call them "flawed." A (properly-implemented) mutable memory structure will have better performance characteristics by definition.

Don't get me wrong, I like immutable data and I use it over mutable whenever possible (Clojure is my favorite language), but to call it "flawed" is an overreach at best.

Mutable data structures are a very accurate abstraction of how computers work. They are a very inaccurate abstraction of how the world works. At any given instant, the world is constant; time moves forward, and changing the world as it was 1 second ago is not possible. Insofar as most of business programming is trying to efficiently give answers about the state of the world at various points in time, I would agree with gcv that "flawed" is the appropriate term to use for mutable data structures.

> They are a very inaccurate abstraction of how the world works.

No, they are only inaccurate software abstractions of the business abstractions they are supposed to model. People like immutability and reversibility of concepts and that's why they try to apply it in business, law, finance etc.

But the real/natural world, at least from our subjective perspective is not like that. Things in the world are not practically reversible or immutable - you cannot "restore" a deadbody. The real world is loosy and mutable, at least from an agent's perspective. The human brain is so efficient at what it does simply because it just "throws way* (irreversibly!) so much information every second, filtering for just what it needs. A small corner of the world is our "business/finance world" and using immutability and reversibility is great here, but there's way more to computing than implementing business/finances logic and processes in software!.

...if you think about it, classical OOP is inspired from cellular biology (can't find the reference now, some musing of Alan Kay I think...), so it's no wonder why it's such a trouble maker when used to model business logic, it was the wrong tool for the job from the get go. But generalizing that mutable data structures are a bad tool to model and understand the world in general is waaaay overstretched, the world is bigger than "business", and all modeling approaches are equally useful.

Unless I'm missing the point of your argument, it sounds like you're saying that, e.g., computational biology does not benefit from representation as immutable data structures. Although I have no experience writing software for modeling cell or biological entity behavior, I strongly suspect that reasoning about the simulation itself by representing its states through immutable data structures, or immutable databases, will lend itself to a better piece of simulation software.

If you think I'm wrong about this, I'd appreciate some examples.

Mutability is change without chronology, which isn't the way the real world works. Immutability over time is a more accurate abstraction of physical processes, particularly when you get into smaller time increments and services distributed over larger distances.

Mutability is useful as an optimisation, but I don't think it has much worth as a model.

>The real world is loosy and mutable, at least from an agent's perspective.

No, it's not. You cannot mutate any fact! You will never be able to change the fact that Hitler existed, for example. Facts never change.

This is the purpose of Datomic: to store facts. Mutable databases only seem to be interested in maintaining the most recent fact about any particular piece of information.

Your comment is not a response to the comment you replied to. In particular, it was quite clear about the flaw in question:

> Mutable state makes reasoning about program behavior more difficult than necessary.

You could dispute that claim, but bringing up performance characteristics is totally irrelevant to it.

Mutable data structures are a much more accurate abstraction when it comes to how computers actually store data in memory and on disc

Yet programming in imperative, mutating, mainstream languages is nothing like how code works at the machine level. I understand you're talking about data, but I think the analogy still stands. Or maybe a better one is that a file system or even a database itself is an abstraction of how data is stored on disc.

I see the created_at and updated_at fields as the poor man's datomic.

Besides what has already been said, I am interested because it is one of few distributed, consistent databases capable of transactions. Google Spanner, Hyperdex and Postgres-XC are the others I know of; they appear to be quite a rare breed.

There are other distributed consistent databases (HBase, RethinkDB, arguably Cassandra if you always use QUORUM, etc.) but neither supports transactions or has a design that seems to me like it easily could.

Is it distributed nowadays? I vaguely remember Datomic as having a single-point-of-failure "transactor" which every request had to go through. (Given the specialized name, maybe the transactor itself doesn't have to do very much; but it still makes me wonder how you'd distribute the data store over many nodes if the nodes aren't capable of transactions themselves.)

Writes have to go through the transactor. Reads don't. So Datomic is a bit of a hybrid; writes are single point, reads are distributed.

The storage layer is distributed automatically because it's decoupled from both the transactor and the peer. It's actually pluggable, too... you can use DynamoDB, Riak, Couchbase, Infinispan, Postgres, etc. which all have different performance and availability characteristics.

I'm ok with serialised writes for most purposes.

Every time datomic is brought up, people always chime in with this whining about it being closed source and I just find it sounds entitled and petty. It makes it seem like with enough people agreeing with you and enough upvotes or whatever, you think the releasers of datomic might reconsider and open source their code.

This is the same person/people that released clojure to the world (as open source). It's not like they're a company with a history of closed proprietary software, they are fully aware of their options, and I'm sure they have their reasons for keeping it closed. Aren't they entitled to do whatever they want with the code they produce without frowny faces all over the comment threads?

DBs are generally a hugely crucial part of your infrastructure, and it makes sense to be conservative with them. Part of that conservatism is a heavy bias toward open source. Especially given a heavily idiosyncratic database like datomic, with large switching costs to anything else, relying on a proprietary product that could disappear at any time is a huge risk factor for your business.

This is why the business model of nearly every NoSQL business is to provide commercial support for their open source DB. Examples include Basho (Riak), Datastax (Cassandra), 10gen (MongoDB), RethinkDB, VoltDB and Couchbase. There are a few exceptions (mostly in the very large data space), but the general rule is: if you're promoting a new database, make it open source and sell services/enhancements around it.

In my experience, corporations and most programmers love closed-source software and services. Like Amazon AWS (and everything in it like S3, DynamoDB, etc), GitHub, MacOS X, whatever. But if it's free, they'll certainly freeload on that too. There are advantages to the opensource strategy, but there's tradeoffs.

Programming languages are different; Clojure likely wouldn't have succeeded were it closed-source.

(BTW, Basho and Datastax seem to have "enterprise" products. Are these closed-source or contain hidden closed-services? And we should keep in mind that DBs like Riak and Hadoop are clones of closed-source DBs. Basho employees even mention that the authors of the Dynamo paper were probably nearly fired for releasing it.)

Databases tend to fall closer to the programming language side of the spectrum, in my experience. If Github shut down, it would be a hassle but not a fatal one. Same goes for most of AWS (DynamoDB and S3 have compatible open-source alternatives if need be).

But if your esoteric programming language or DB (I place datomic in that category because it's very different than anything else out there) becomes unavailable or the price is raised too high you have no choice but to fundamentally rewrite your software. That could be the end of your business.

This is absolutely not to say they should open source it--that's entirely their decision. This is just to explain why many (myself included) feel uncomfortable relying on closed source DBs.

Yes, that's a deep conflict in the tech world, even beyond this topic. For example, does a manager select PHP because PHP programmers are commodities? Some managers tell me that's how they select technologies: how commoditized its labor supply is. (They consciously accept the price of technological conservatism, to avoid real risks.)

I personally think it's important to weigh the various risks. Keeping in mind there are "opportunity costs". So for example, if I were to choose something like Datomic, it's almost certainly because I predict much greater productivity than alternatives. There are risks to giving that up, which I must consider.

And there are ways to mitigate some risk. Like simply asking what happens if they go out of business. Much like investigating potential failure modes.

I think it has a lot to do with whether you're primarily aiming for new functionality or for reliability. Clojure, GitHub, AWS, etc. are primarily used because of their extra functionality, and downtime/failure is often inconvenient but not disastrous.

Compare this to a database-reliability is tantamount. If there's a weird bug you need to ensure you have amazing support, you have access to dig into the internals and solve the problem yourself, or both.

Datomic comes across as prioritizing functionality over support, e.g. they're closed Source and don't (yet?) have the reputation for Enterprise class support businesses usually look for with closed source DBs. That doesn't mean they'll fail, but I'm curious to see who Datomic's customers will be.

Yes, we could both compensate people for their work, and have it totally open. It's certainly imaginable; it doesn't violate a law of physics. But that would require collective action, like public subsidy. While people refuse to work towards that, for whatever ideological reasons, then we won't have it both ways.

Actually, it could be done with crowdfunding.

Shameless plug: https://bountyoss.com/

All the code the company I work for writes is AGPL licensed, and we don't need rent-seeking^Wpublic subsidies to get compensated for it. The trick is to get an agreement before the code is written, as well as providing other services around that code.

They are entitled to release however they want. We are entitled to respond how we see fit. Who are you to say what our response should be.

The OP didn't say they ought to open source it, just that they cannot justify using it because it is closed source. There is a huge difference.

Aren't they entitled to do whatever they want with the code they produce without frowny faces all over the comment threads?

No, why should they? Nobody is entitled to be free from criticism.

No, why should they? Nobody is entitled to be free from criticism.

Can you really be critical of someone for choosing to close source their own code?

Well, it wouldn't make much sense to criticize them if they had kept it on their hard drive (though people criticize others for what they do in their own bedroom), but as soon as they release it to others, it's a social act, subject to praise and criticism just like any other.

Another 'broken on mobile' site (Aurora on Android here).

The tech might be cool, the site is awful.

Downvoters: I like Datomic, like Stuart, but the site is crap. First visit had lots of artifacts (content was seemingly above the annoying header), cannot reproduce that now. Right now a huge part of the display is a fixed header, the content is 'unzoomable'. There's no way to say anything nice about that sort of stuff.

http://db.tt/sIstWJhZ

Looks great on the default android browser.

One of the craptastic Blogger templates. I've taken to viewing them through Google Cache, when possible (and when I'm sufficiently motivated to not simply quit the item/site); unfortunately, more and more seem to be excluding themselves from said caching.

http://webcache.googleusercontent.com/search?q=cache:http%3A...

Hmm. It seems Google Cache doesn't help me here. Now I was able to reproduce the artifacts in the page header, but the huge header itself remains (too much screen space, literally zero value) and the content still fails to allow pinch/zoom to actually read the text in the illustrations.

http://db.tt/Yz5J6zd4

It is a key value proposition of Datomic

Is that a purposeful pun? Because it confused me.

Reply on news.ycombinator.com