What does the 4.0 api look like? I would home that they would expose a high-level interface for change data, similar to SQL Server or Snowflake streams. Most databases leave this as an afterthought, which is so unfortunate.
From Netflix: https://www.datastax.com/accelerate/agenda?session=Live-Traf...
From Yelp: https://www.datastax.com/accelerate/agenda?session=Streaming...
From DataStax: https://www.datastax.com/accelerate/agenda?session=Change-Da...
Videos are already published: https://www.datastax.com/accelerate/sessions - you just need to select correct track...
The high-level idea is hard-linking the commit log segment file in cdc directory as it is being written + have a separate index file to keep track of offset.
What happens if you write a row to Cassandra but that row is invalidated because of failed writes to the other nodes in the cluster (ex. the write was QUORUM and it succeeded on the node that you're reading from but failed on all other nodes). In Cassandra, when an error occurs, there are no rollbacks, and the row gets read-repaired. It sounds like that row will get outputted to Kafka. How do you "disappear" data written into Kafka if it gets read-repaired?
This is why it is recommended to always retry when a write fails. This way data might arrive at BQ slightly before cassandra itself for a failed write, but eventually they will be consistent.
That said, at WePay cassandra is being used by services which do not have to be strongly consistent at the moment. Not to say it’s not possible, as there are companies that use Cassandra’s light weight transaction feature for CAS operations.
As for consistency between kafka and cassandra, agree that its hard to make it 100% consistent (due to a number of limitations cassandra poses). This is why we tend to be pretty careful about which services should use cassandra, and specify clearly what is the expectation and SLA for cassandra cdc.
select * from changes(tablex)