Does WarpStream guarantee correct order inside partition only for acknowledged messages or also among the acknowledged messages (in different batches)? If so how do you keep clocks synchronized between the agents?
It guarantees correct ordering inside a partition for all acknowledged messages regardless of which batch they originated from. We don't synchronize clocks, the agents call out to our cloud metadata store which runs a per-cluster metadata store that assigns offsets to messages at commit time. Effectively "committing" data involves two steps:
1. Write a file to S3 2. "Commit" that file to the metadata store which will then assign the partitions at commit time 3. Return the assigned partitions to the client