Python – Writing large ZIP archives without memory inflation

Sounds like you need this... a previous submission as linked elsewhere mentioned https://github.com/longaccess/python-zipstream/tree/streamin... which should do it.

Edit: Dang it, GPLv3, sorry (vs Apache 2 of your project).

The two previous discussions with comments may offer enough to get you what you need: https://news.ycombinator.com/item?id=23198233 https://news.ycombinator.com/item?id=25616513

Edit 2: https://github.com/kbbdy/zipstream, no license (yet)

I was thinking the same. Unless you're doing something really wrong, or if you are intentionally writing to a memory buffer, memory consumption of ZIP compression should be minimal across the whole process.

You do need to keep the central directory record to be written at the very end, so it might cause a concern if you have a lot of (probably small) files. But Zipfly doesn't seem to solve that anyway.

So on one top-level comment we have someone arguing about just writing the entire file to disk and don't bothering; while on the other top-level comment someone argues that the zip algorithm should already be using constant memory anyway.

Funny to see the usual great programming school divide so succinctly put :)

Anyway, already seen a number of times https://news.ycombinator.com/from?site=github.com/buzonio

ZIP in itself can have different compression algorithms. deflate and "no compression" (store) being the most common by far (because compatibility reasons), followed by deflate64 and and a long tail of things like bzip2, zstd, lzma and others (the ones I mentioned are some of the methods "officially" assigned a method number by pkware). A lot of software uses zlib to implement zip, and there the only real options for broad compatibility are therefore deflate and no compression. deflate has a small fixed-size window buffer, with no compression you can avoid such a buffer altogether. No compression is quite alright if you put data into the archive that cannot be compressed efficiently anyway, like already compressed multi-media formats, and it will save you and whoever uses the archive later a bunch of compute.

The only thing that may eat up memory is the final name directory in the "footer".

No compression also has the added benefit that you can pre-compute the final size of the zip file if you know what goes into the zip already (it's just a matter of summing up the sizes, and the sizes of the info entries and final footer entries), which can be nice when sending the result over http, as it enables you to send a Content-Length, instead of not sending one because you don't know it yet, using chunked transfer encoding and letting the browser show some "20MB of unknown downloaded".

Then it really depends on the implementation. A lot of libraries will buffer in an entire input file, compress it in one go and store the result back to disk or some buffer, and things like that.

But it is perfectly possible to implement a zip writer that implements a stream/callback interface to read the result compatible with whatever your language is using (e.g. C++ istream (yeahyeah :P), C# Stream, python file-like objects, node Readable streams, etc).

With the python standard library zipfile implementation, while it is possible to open "file-like objects" (i.e. objects implementing read/write/seek, like a BytesIO buffer), it isn't really possible easily to drain the data after just a chunk of it is done (or have it output to network/http streams, as these usually do not or cannot implement seek). The OP implementation solves this by defining everything you want in the zip beforehand and then using python generators to get the data out in chunks that you can then write to whereever you want sequentially.

It would only block if you did it synchronously.

If I had to build an endpoint that generated ZIPs on the fly, I would explicitly build it to be asynchronous and efficient as possible.

The only issue is that you might want to attach a buffer somewhere that might dump to disk past a certain amount, if the source of your data can’t be “paused.”