About the main reason I haven't pushed on using it for work is just concerns about maturity. gzip/pigz works well enough for our purposes, and isn't enough of a hindrance to justify using what is still new software without a proven history.
About the main reason I haven't pushed on using it for work is just concerns about maturity. gzip/pigz works well enough for our purposes, and isn't enough of a hindrance to justify using what is still new software without a proven history.
To that end, every release (and pretty much every commit) undergoes significant testing. We also fuzz test heavily and have significant regression tests in place constantly looking at corner cases and verifying known prior bugs stay fixed. It's impossible to prove there are no issues, but so far we've put a lot of effort into being as reliable as possible and seen that play out well in our production environments. We trust it.
I just like to be careful with a mental "new tech" budget for a project/platform. The more new tech is involved, the more risk there is. The balancing point there being the value the new tech brings.
With the platform I'm working on at the moment, the gains are relatively minor: shaving a few seconds off here and there, when things are in the scale of minutes; and saving 20-30MB when things are in the scale of gigabytes, but also not lots of them. As the project grows, and as the new tech we've introduced proves itself, there's always scope to introduce other new tech like Zstandard and start gaining in the storage savings.
As much as I trust Yann, I probably wouldn't use Zstd for anything mission-critical yet.