Bitemporal History | Hacker News Reader

Bitemporal History

martinfowler.com

64 pointsr4um19 comments

As we are working on complex trading systems it is frequently necessary to correct past record and then be able to make consistent reports based on "as of" date. This means answering questions like "I want to get the state of this population of trades as it was yesterday at 5 pm but based on our knowledge yesterday at 10 pm."

Pretty fun actually when you have billions of these records:)

Shout out to Crux (https://opencrux.com/), a bitemporal database. I've been using it for a year now and it's fantastic.

Why would you need a special database to implement this feature? Usually its just modelled by having 2 date fields and querying as appropriate.

You can implement bitemporality on top of a database kernel not explicitly designed to support it but you sacrifice significant performance and scalability. It is a bit like trying to implement a graph database on top of a relational database -- it definitely works, just not well.

Database kernels designed for first-class bitemporality support have low-level internal structures to optimize operations on bitemporal data models not found in typical database kernels. These features are not free if they aren't being used, hence why they are relatively rare in databases that are not explicitly bitemporal.

Crux timestamps sit outside of the data model and apply universally across the whole database (the data model is documents not tables), so you don't have to consciously think about managing bitemporality at all by default, but whenever you might need it, it's there. This means you don't have to adapt your queries to account for extra fields/columns, instead you just specify the timestamps of interest as simple parameters to the query datasource.

Also, Crux maintains a point-in-time (a.k.a timeslice) temporal index which is really a Z-Order Curve[0] that stores the two dimensions for fast lookups in a local KV store like RocksDB or LMDB. By contrast, a userland RDBMS implementation of bitemporality will be significantly slower to execute similar point-in-time "as-of" queries.

[0] https://en.m.wikipedia.org/wiki/Z-order_curve

(I work on Crux)

Check out our framework for bitemporlity: https://github.com/scalegenius/pg_bitemporal We use it in production for 5 years. I am ready to answer any performance/indexing and other questions

I have not read the article, but I have written a bi-temporal ORM, and I find it to be such a useful concept in practice. One of the primary benefits is optimistic locking. You can say goodbye to explicitly locking during a db transaction.

This sounds interesting, could you please share more details about it? (or the source code if available)

Not the OP, but Reladomo is a Java ORM which supports bitemporal history.

https://github.com/goldmansachs/reladomo

I ran across this notion for the first time when I started a job in finance a few years back and as an interview question they asked me about it. I hadn't heard of the notion and so afterwards tried to find some good resources online to read up about it.

What I found was not great. So glad to see this idea getting a good write up.

Reminds me of rebasing Git branches in order to clean up commit histories, or rewriting Git repositories in order to change usernames/emails stored inside.

I just hope that whatever systems get created (bitemporal history, version control, etc.) have support for "replacing" previous usernames, or deleting events prior to a specific "record time" and replacing them with newer understandings of history. Doing so in Git (a more-or-less append-only system) results in multiple histories which diverge at the moment of the change.

In simpler cases, you can see the problem as purely functional data structures.

And for the rest of us there's always ConcurrentSkipListMaps.

More specifically, retroactive data structures[0] are an interesting lens for thinking about bitemporality. After watching some great Erik Demaine lectures on temporal data structures I wrote a bit about the relationship to bitemporality on the Crux site[1]. (Crux being a bitemporal database with graph-friendly Datalog queries)

> In summary, the Crux indexes as a whole could be described as a "partially persistent and fully retroactive data structure".

[0] https://en.wikipedia.org/wiki/Retroactive_data_structure

[1] https://opencrux.com/articles/bitemporality.html#_retroactiv...

Related to bitemporality, I'd be curious to hear whether anyone has experience with the Temporal Tables[1] extension in PostgreSQL?

[1] - https://github.com/arkhipov/temporal_tables

I haven't used `temporal_tables` but I have built similar temporal systems on Postgres in the past. (Judging by https://github.com/arkhipov/temporal_tables#usage at least.)

The trouble with these sorts of approaches is that they solve temporality the same way we did in the 90s: Add an `entity_history` table, timestamp your tx-time and valid-time, and add a trigger to version your entities. This must be done for each entity you want to version across your temporal plane.

It would appear that `temporal_tables` doesn't support bitemporality yet. It only has tx-time (system time). But even if it did, this approach doesn't help you with live data. Because the `entity` table corresponding to the `entity_history` table still permits destructive updates, temporal queries are always in the realm of audits and can't answer questions about the application data directly. Add to that a completely manual system of querying the temporal information, and the resulting systems tend to get quite hairy, which is why Martin recommends avoiding bitemporality whenever you can. Unfortunately, that recommendation (while sound, for relational databases) means that bitemporality is expensive and manual if and when it's implemented.

A bitemporal database like Crux encodes the temporal plane into all the data stored in it, making it transparent to the user. There's no up-front setup cost to bitemporality and a query's default time on both time axes is "now", allowing the user to ignore temporality entirely except in those few instances where it is required -- but when it is required, it is global.

Sure, I can appreciate that native support for a feature like this is nice.

As I understand it, most implementations (including another one for bitemporality[1]) involve either audit tables, as you mention, and/or additional support columns. It's as if the "now" representation is simply a narrowed view within the full set of underlying, (bi)temporal data.

That said, PostgreSQL encodes and has battle-tested decades of database functionality, including a surrounding ecosystem, so I'd be a little wary of switching technology even if it does solve one individual problem thoroughly. Everything has to start somewhere, though.

[1] - https://github.com/scalegenius/pg_bitemporal

Perhaps this was an overly-skeptical perspective. Datalog (as featured in Crux) has certainly existed as a concept for a long time, although it hasn't (as far as I can tell) seen popular uptake.

The SQL Standard actually specifies two related but separate approaches to versioning:

- system-versioned tables, which is the focus of this extension. This can be implemented purely database-side, and the application doesn't even need to know whether (certain) tables are system-versioned. This is mainly intended to offer change tracking and auditing in a standard way, to replace all the home-grown solutions to do the same.

- application-versioned tables, which I think Postgres already supports natively. This puts the application in control of the timespans in which a row is considered valid, and is probably what you would use for retroactive (or planned) updates to business records.

I'm not sure if the standard specifies how these two versioning systems interact, but by combining both strategies, you could in theory have a full system-versioned record of who made which changes to take effect on which date.

Reply on news.ycombinator.com