zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems.
I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json
I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json
Writing your files directly into a compressed stream and decompressing on the fly has become almost a standard workflow for any files I'm going to read and write sequentially anyways. No need for the data to ever exist uncompressed on the file system. Previous formats never did that for me because they either had too much overhead or too little gain, often both
Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults.
gzip 7.3% 21s 9s
bzip2 4.6% 251s 50s
bzip3 3.3% 82s 69s
zstd 6.9% 2s 3s
lzma 4.7% 51s 3sUnder most r/w workloads, using parquet/lance/vortex/native-duckdb, with their built-in columnar compression will result in more performance AND space savings. Non-solid compression. Then, the query engine can push down your query predicate to a column row group level, instead of forcing it to decompress the entire dataset to operate.
Practical example: duckdb has syntax - https://duckdb.org/docs/lts/data/multiple_files/overview - to glob multiple files at once, but that really only works if you're applying push down query predicates instead of re-decompressing your entire data set per SELECT. I would say for most dataset, even 20%+ size is worth not having to decompress (or even download!) the entire dataset, to figure out if something fits the predicate.
After all, if you have to download and decompress the dataset back again to operate, then the "space savings" are gone.
It's hard to understand what point you're trying to make. Can you clarify?
A go-to thing means it's a sensible default choice and has no little to no downsides (versus not using compression), it doesn't mean it's the best for everything.
Until now the go-to has been DEFLATE (gzip and zip) but zstd is definitely competing against it because it is better in almost every way.
Since we kind of need a default "Need to compress something? Use this!" setting - would you prefer zlib over zstd, or something else for that role?
If your platform/sdk/browser/standard-library comes with zlib/gzip/.. then it's often easier to just pick that.
No new dependencies is always a win. App size. Security, etc.
Otherwise, if zstd is easy to add, IMO I would always prefer, zstd, lz4 or brotli.