Hacker News Reader

Comment by majke | Hacker News Reader

I'll use the opportunity to moan about the state of packaging in the Python world. Why, after so many years, there is no way for me to ship software written in python, in deb format?

I end up doing some real crazy stuff:

* build in docker

* using virtualenv /usr/local/mypackage

* then manually copy over the init scripts "cp /usr/local/mypackage/bin/* /usr/local/bin"

* and debianize "/usr/local/mypackage/" and "/usr/local/bin".

Except it doesn't work:

* I need root do create the venv in /usr/local.

* This technique copies garbage like "activate" script to bin.

* virtualenv ships its own python binary. If the python version on host differ, it won't work.

* The workaround is to rewrite the bin scripts.

Total mess. All I need is a deb with some bin scripts, and all dependencies bundled. Is it _that_ hard...

View on HN

I tried a few things myself, like fpm[1], rpmvenv[2] and lately pex[3] - but so far, I haven't found something, that I would call solid and simple. We have constrained deployment environments that do not allow most of the tools, that would ease the process on a development machine. Basically, I need a deb or rpm.

I have some hopes to wrap a pex into deb/rpm, but I would not call this approach simple.

That's unfortunate since Python is a wonderful language for many data-sciency tasks - Python makes things possible and pleasant, that would be a pain in other languages.

* [1] https://github.com/jordansissel/fpm

* [2] https://github.com/kevinconway/rpmvenv

* [3] https://pex.readthedocs.org/en/latest/

* Another tool: https://github.com/spotify/dh-virtualenv

We use dh-virtualenv and it's working out pretty well for us so far.

crohr's pkgr (pkgr.io) works well as well.

> Why, after so many years, there is no way for me to ship software written in python, in deb format?

How about using dh_python? And if you're building in docker anyway, why use virtualenv?

> All I need is a deb with some bin scripts, and all dependencies bundled.

Ah, that would be a problem. Python .debs are supposed to depend on separate python module packages, not bundle everything.

For what you're doing, have you considered turning your entire program into a single executable Python zipfile? See youtube-dl for an example.

You can force virtualenv to include a specific python version

virtualenv -p python2.7 env_name or virtualenv --python=/some/path/to/python env_name

You've made the best argument for using pip like this, however. Instead of having to install a virtualenvironment and install dependencies and all of that, you could just ship your program with the pip -t libraries, though you'd have to include the path somehow on the install so python knew where to find those files.

Well, your user might have python in /some/path/to/python. My user my have in /usr/bin/python or maybe /usr/local/bin/python.

I don't know off hand if this handles dependencies, but I've used this in the past to create deb files:

    python setup.py --command-packages=stdeb.command bdist_deb

I'm pretty sure this works without requiring some special module.

What about dependencies? Setuptools assume all is in place. That's why you need pip/virtualenv. So with that you expect your host to have all the stuff.

Virtualenv installs some .so files in its thing, so venv directory is tightly coupled with the python binary it runs. Ie: you need to sort it out on the moment you build venv, and shipping venv directory is not correct.

I know that distutils supports to create RPM packages https://docs.python.org/2/distutils/builtdist.html#creating-...

I can't see why that cannot be adapted to .deb packages as well...

Did you try conda?

Reply on news.ycombinator.com