Minimizing S3 API costs with distributed MMAP
warpstream.com
warpstream.com
> Compacting the data may sound expensive, but in practice it's highly efficient since it only needs to happen once
Is there any handling for tombstone records?
WarpStream doesn't implement compacted topics today. It is on our roadmap, though. Compacted topics are typically not used in high-throughput workloads, so our plan is to delay compactions for longer than a disk-based system would to trade space amplification for write amplification.
> Compacted topics are typically not used in high-throughput workloads
TIL, but it makes sense. Compaction/retention policies certainly introduce a lot of extra tradeoffs dimensions.