Hacker News Reader

Comment by hellofunk | Hacker News Reader

> one of Datomic's primary features, i.e. having the database reside in memory with your application

I'm not sure I follow - are you saying it is desired to have an entire database also in memory? You don't probably mean that, but I thought a key feature of databases is for them to be your offline storage so things don't need to be in memory. I wouldn't be able to fit the contents of my SQL databases into memory on any machine I own.

View on HN

Datomic keeps your database in the same memory space as your application code, backed by SSD, backed by EFS, backed by S3/DynamoDB. Each layer of this cache provides a different value, and the whole cache system is automatic and eternally consistent because the database is an immutable value.

You want the illusion that your database is entirely in memory, just as want the illusion that all your memory is in your L1 cache.

> Each layer of this cache provides a different value

What does that mean?

And what kind of actual memory usage should people expect with a "big" database?

Hm, I should have said "value proposition", e.g. SSDs are low latency but can fail, S3 is high latency but "never" fails, etc.

Datomic is designed for data sets larger than memory, and also to do a good job caching working sets that do fit in memory.

> Datomic is designed for data sets larger than memory

Of course. But you didn't answer my question.

Let's say someone has an application with 100GB of data in Datomic.

What kind of memory usage should he expect for the "peer" process?

That depends on what queries the peer runs. In both On-Prem and Cloud, Datomic maintains an in-memory LRU object cache. Also, you can send queries with different working sets to different processes without having to shard data on ingest. For example, some peers might handle queries related to user transactions while other peers handle analytic or batch work.

> You want the illusion that your database is entirely in memory

I don't want illusions. When it comes to databases I want simple and predictable.

Datomic basically does the same thing your database does - caching the working set in memory and running the queries. The difference is that with datomic the db is running in your java process rather than on another server. This should get you better scalability (for reads) and make complex queries faster. (Writes still go through a designated server.)

The data is modeled as a collection of (entity,property,values,txn) tuples (with schema on property). They are stored in a log and a few ordered sets (indexes) that are structured somewhat like 2-layer btrees with large pages. These pages and the tail of the log are loaded as needed while running queries. The nature of the data model also gets you point in time queries and queries across time (e.g. all previous values of a property).

There is some cleverness going on behind the scenes, so for a large database you wouldn't necessarily have the whole thing in memory at once. But one of the big selling points of Datomic is that you get to treat the entire database as a value, and when you configure your app as a Datomic Peer (the feature I referred to above), it loads most of the Database into memory as it is needed.

I think the idea of databases being on-disk (vs in-memory) is getting quite outdated. You want durability of course, but most datasets can fit in memory these days.

(Which is natural if you consider the retarded development of disk/ram vs processing power; cf the memes of "disk is the new tape" "ram is the new disk")

Reply on news.ycombinator.com