There were various "archive" ZIP files that had grown to several gigabytes each. The developer was using Info-ZIP "zip.exe" to "append" files to the archives multiple times per hour (>100). Like virtually every ZIP implementation, Info-ZIP just re-writes the entire file when appending. As a result, I was seeing multiple terabytes of I/O churn each day as new ZIP files were being created to replace the old ones (which was killing VM replication across a 500Mbps pipe, and is what drew my attention to begin with).
While I would have loved to find a ZIP implementation that just appended the file and a new central directory I found it easier to just rename the ZIP files every day to minimize the churn. Still, I can't believe there isn't a ZIP implementation out there that can do this.