Vector clocks I typically want to stay away from as far as possible.
What do you do during a network partition? Accept writes that you’ll throw away eventually?
Imagine a customer has £100 in their account. System partitions. Customer withdraws £70, hitting one DC. Customer then hits the second DC, this time withdrawing £50. Each DC thinks the transaction is valid, and so serves it.
Later when the partition is restored, events are played back, and divergent history is detected via the vector clocks - the two withdrawals are not causally related. Remediative action can then be taken.
Transactions prevent bad things happening, but require CP semantics. Eventual consistency allows AP, allows bad things to happen, meaning you have to be able to detect them and clean them up later.
> Remediative action can then be taken
Sounds expensive and error-prone; taking a “read only” outage makes more sense in many use cases
---
I've not really used Kafka, so couldn't comment on that. I did some work for a customer that involved multi-DC microservices with isolated databases (ie one DB per DC). We used event sourcing with vector clocks to do manual reconciliation of the databases including during partition. Reconciliation involved custom logic depending on the event type, so not sure how a transport mechanism like Kafka would handle that.
A talk about the approach is here: https://youtu.be/hYVh8PbbeJw
I get to vector clocks at the end.
Edit: to add on about Kafka, it guarantees a serializable ordering on the incoming messages and is driven internally by a vector clock. This may not be suitable for applications, but the throughput is high and allows multiple subscribers to get a consistent ordering.