Hacker News Reader

Comment by kieckerjan | Hacker News Reader

Zstandard is great indeed. Performance is excellent, but in my experience its killer feature is the user definable dictionary. My search engine software stores its documents in an archive to which I need random access. Lots of small documents, random access: bad for compression. Use of zstd with a tuned dictionary improved compression fivefold(!) when compared to the older zlib solution.

View on HN

> but in my experience its killer feature is the user definable dictionary.

If I had to guess, since this originated from within Facebook, they're using the feature for user definable dictionary to maintain a large corpus of sample, representative user uploaded content.

Which would accomplish the purpose of tuning its performance for their own specific internal usage.

This didn't originate from within Facebook. It is an evolution of the work done in late 2013/early 2014 around FSE[1]. Facebook later hired Yann to support his work around this (pretty amazing!) tool.

The ability to define your own dictionary is useful beyond simply large indexable datasets. Anything application-specific can usually benefit from user-defined dictionaries.

[1] http://fastcompression.blogspot.com/2013/12/finite-state-ent...

The dictionary helps with the boilerplate stuff that is shared between records but not repeated inside them. So it works like a charm for relatively small fragments of structured data like e.g. HTML pages, XML-docs or SQL statements: lots of "SELECT * FROM" fragments that can be replaced by a single token. I suspect it works even better for this kind of data than for natural language data, which tends to be skewed in a Zipf-like way.

I've wanted to use zstandard with a dictionary, but how do people manage their dictionaries? What if I want to update a dictionary, will that break old compressed data? Do I need to 'tag' the data so I know which dictionary to fetch?

It is not the responsibility of a compressor to manage (or even store) your data. So yes, you have to come up with some sort of scheme yourself. But the overhead of a version number or something is usually negligible compared to the gains.

I'm a lot less concerned with adding a version number and much more concerned with the additional complexity of having to fetch a dictionary.

We documented some of this in our Engineering blog posts on Zstandard. Check this one out and search for "Managed Compression" for the system we built.

https://engineering.fb.com/2018/12/19/core-data/zstandard/

My first thought would be to store the dictionary in a file with the same name as the archive, but with '.dict' or something like that appended. If you're concerned about wasting space by duplicating the dictionary for many many archives, use a single file and hard links instead.

Reply on news.ycombinator.com