https://blog.cloudflare.com/squeezing-the-firehose/ is our story of how we moved from lz4 to zstd (with a stop at snappy in between) in our kafka deployments. Results are/were similar to what Heap is reporting here.