Hacker News Reader

Can someone explain what it's doing? Is it using an algorithm with far superior space complexity than the usual algorithm?

Python seems a curious choice. Compression is computationally intensive.

View on HN

Looks like it just splits by 16MB chunks, so just standard deflate. Actual compression is handled by the python zipfile module, which is probably C code underneath.

zipfile uses zlib which is C. But it's even better than that: it releases the GIL, so it gives linear speedup with multiple threads. If you need to (de)compress a bunch of files, you can do them all at once quite easily using e.g. concurrent.futures.

If you want that speedup on the command line without Python, check out pigz. It's gzip with parallelism. Easy 10-20x speedup for some jobs.

Note that pigz only parallelizes compression, not decompression.

Reply on news.ycombinator.com