Datomic Implicit Partitions

Datomic uses any of a number of stores, typically something like DynamoDB, to persist this index data.

When performing a query, Datomic peers read directly from the store (or from a cache) and read an entire block of index data as a single unit - each such block effectively contains many entity/attribute/value/transaction Datoms sorted by entity, attribute, or value, depending on which index is being scanned.

In other words, when reading all the attributes for entity X, Datomic will read from storage the block containing X, but also must inadvertently read other data that precedes or follows X.

The partitioning business is to ensure that related entities are stored close to each other; literally, a block of bits within the 64 bit entity id is set to the partition value. This ensures that the Datoms for related entities are stored close to each other, which in turn, ensures that the query can be satisfied using fewer of these index blocks, read from cache or storage, than if the entire entity id was arbitrary. Essentially, if entity X and entity Y are related, they can share a partition, and most likely will be stored together in a single index block.

This co-location of data can make a big difference in very large Datomic databases.