All the Ways to Compress and Archive Files in Python

towardsdatascience.com

11 points·dazzling_nash··6 comments

Only tangential, but it's a very little know fact that python can import and even run code from a zip archive. There is actually a module for that in the stdlib: zipapp.

This means that you can zip your entire project, including deps, into a pyz file, and ship that to your serveur as one big archive and run it like you would a .py file. It makes deployments very easy.

It also means you don't have to worry about using dependancies in quick scripts: use as many as you want and bundle them!

A tool exists to create the bundle easily and make it extract itself on first run for better perfs:

https://shiv.readthedocs.io

Is that instead of the zipapp module.

Does zipapp automatically bundle dependencies, or do you need to specify that somehow? Also, how does this handle native extensions?

Edit: Answered by first question: https://docs.python.org/3/library/zipapp.html#creating-stand...

for bundled deps, see shiv or pex.

https://shiv.readthedocs.io/en/latest/

https://pex.readthedocs.io/en/v2.1.49/

You still need the host to have a base python interpreter.

Carefull though, pex doesn't work on windows

pip install --target to download your deps then use shiv

The only viable compression library, for internal use, is Zstandard

https://python-zstandard.readthedocs.io/en/latest/

https://engineering.fb.com/2016/08/31/core-data/smaller-and-...

The compression ratio and decompression speed just blow everything else away.

Reply on news.ycombinator.com