Hacker News Reader

I have a project where I want two properties which are not inherently contradictory, but don't seem to be available together:

1. Huge compression window (like 100+MB, so "chunking" won't work)

2. Random seeking into compressed payload

Anyone know of any projects that can provide both of these at once?

View on HN

If seeking to frames is good enough for random seeking, this can be done with zeekstd already. The cli sets a custom window log (ZSTD_c_windowLog) on the compression context when creating binary patches[1], I regularly use it with a window size above 1G.

[1] https://github.com/rorosen/zeekstd/blob/main/cli/src/compres...

Nice, thanks.

That sounds like this with inline dictionaries added unless I’ve misunderstood you.

Reply on news.ycombinator.com