This might not be needed if this is strictly used for FIFO event consumption, but I guess I was thinking of trying to make a system like this support time-sliced queries.
This might not be needed if this is strictly used for FIFO event consumption, but I guess I was thinking of trying to make a system like this support time-sliced queries.
A log gives you "transaction time" but you need to create an efficient representation of "valid time" for backfilling and corrections.
[1] https://en.wikipedia.org/wiki/Temporal_database
Disclosure: I work on a database for Kafka that provides point-in-time bitemporal Datalog queries https://github.com/juxt/crux
For the "valid time" primitive I was thinking of implementing something like a hybrid logical clock that CockroachDB has (but with looser guarantees, mostly just need uniqueness and monotonicity). A sequential ID would provide a slightly nicer interface for pagination but has all the problems that I previously mentioned.