Nice work. Indeed, we still can find some open performance issues related to disk reads in the Kafka community. Although there are some solutions, they are not very elegant. Like https://issues.apache.org/jira/browse/KAFKA-7504
For Kafka, the page cache is indeed a double-edged sword.
Very practical approach! I'm curious about its scalability with high data volumes. Are there potential trade-offs or limitations?
When Kafka deals with large volumes of data, scaling out or in requires large data replication of the partitions, which is a very heavy operation. With the architecture based on S3 using shared storage, everything becomes much simpler. In my view, the trade-off is that we additionally use EBS as a persistent write cache to work with S3, which to some extent eliminates the latency issue of accessing S3.