How To Package Your Python Code | Hacker News Reader

How To Package Your Python Code

scotttorborg.com

157 pointsstorborg34 comments

I've wondered why using a dictionary to initialize setuptools.setup() isn't more advocated in such guides. I understand this is superficial, but imho, this looks much clearer than the author's suggestion [1]:

    from setuptools import setup

    kw = {
        'name'         : 'funniest',
        'version'      : '0.1',
        'description'  : 'The funniest joke in the world',
        'url'          : 'http://github.com/storborg/funniest',
        'author'       : 'Flying Circus',
        'author_email' : 'flyingcircus@example.com',
        'license'      : 'MIT',
        'packages'     : ['funniest'],
        'zip_safe'     : False,
    }

    if __name__ == '__main__':
        setup(**kw)

Here are a few of my setup.py files that follow this convention [2] [3] [4].

[1]: http://www.scotttorborg.com/python-packaging/minimal.html#cr...

[2]: https://github.com/gvalkov/harstats-graphite/blob/master/set...

[3]: https://github.com/gvalkov/jenkins-autojobs/blob/master/setu...

[4]: https://github.com/gvalkov/python-evdev/blob/master/setup.py

Because the only difference there is neater indentation, which you could do using the keywords method.

Additionally, there's no point checking that the script is the main file, as there's only ever one use-case for the script, and that's to call setuptools.

It actually comes in handy if you're using Sphinx and don't want to maintain release and version in conf.py separately from the version in setup.py. There's no reasons why the data should be consumable only by setuptools .

I put the version information inside the package, then anything with access to the package (which both setup.py and sphinx have) can access it - quite useful for doing things like checking the version number from the REPL, for instance.

It can be problematic to import your package from setup.py if you want your dependency specification to be useful.

You can use pkg_resources which comes with setuptools to look up the version number of your package in setup.py:

  import pkg_resources
  pkg_resources.get_distribution("PIL").version

Adding spaces to make things line up is proscribed by the Python Style Guide. Which is entirely optional, but many people follow it.

http://www.python.org/dev/peps/pep-0008/#pet-peeves

I'm so glad Go removes these sort of nitpicky style problems with gofmt. At first I was bothered by how gofmt makes things line up like this, but now I prefer no style guide and a tool that formats stuff for you.

Python has a tool (aptly named "pep8") which provides similar functionality. It doesn't make the changes for you, but it's very specific about what you need to fix to comply with the style guide.

Autopep8 makes the changes for you.

Taking a quick look through this it looks pretty good! I find that beginners have tons of trouble navigating the slightly tricky waters that is packaging in Python, so more resources is great.

One thing I think is missing is to point out not to use setuptools, but to use the less broken, more maintained, drop in replacement distribute (http://guide.python-distribute.org/). That website also has some further information on the packaging ecosystem that's worth flipping through.

Thanks, I'll definitely add that to the "See Also" section.

Can you elaborate on the places where setuptools is broken? I wanted to keep things as simple as possible for the sake of this tutorial, and I think the described constraints should avoid any setuptools bugs.

There are various posts on the distribute/setuptools/distutils mailing list. I don't keep track personally, but I do know that it's pretty widely accepted at this point to use distribute.

Doesn't distribute miss out on a few of setuptools really useful features, like develop installs?

It has develop. I rarely use it, but it's there. AFAIK it is not missing any features that setuptools has but I stick to the simple stuff.

The fact that there is no good, canonical guide for doing this in Python is one of the reasons I moved away from it and toward Ruby. With Ruby I had no problems writing and forking gems using the canonical guide (http://guides.rubygems.org/make-your-own-gem/). With Python, navigating the hell that is packaging is just demoralizing when it comes time to give end users a complicated chunk of code. (Setuptools is truly "a hack on a top of a bad design", not my own words.) I hope for Python's own sake that distribute (http://packages.python.org/distribute/) catches on and people are simply shamed out of using older tools, but since Python already has a problem with getting everyone to migrate code away from old things, I'm not holding my breath.

I'm still not sure how to do certain things "right", e.g., distribute a modified version of somebody else's package with my own package. In Ruby, I can do this with bundler specifying a git repo for my own fork of the gem (and then submit a pull request for the original fork, if it's a patch that's useful upstream).

if setuptools is a hack on top of bad design, distribute has exactly the same issues, since it is simply a fork that was started because of the impression that setuptools dev was stalled. None of the (numerous) design issues from setuptools have been fixed in distribute.

Here are other good packaging guides for Python (by Tarek Ziade). I think Scott Torbog has joined the ranks of these greats:

http://www.aosabook.org/en/packaging.html

http://guide.python-distribute.org/

Are setuptools suitable for packaging the whole web application (for example in Django) together with html/js/css files, configs for uwsgi and some management scripts?

I once tried to do something like this and failed and I had an impression that setuptools are not really intended for such things, but mainly for packaging plain Python modules.

Is this a right impression? If yes, what is a good way to package the whole Python based web application?

It sounds like you're trying to package an entire application, not a python package. If thats the case, maintaining and distributing it may be simplest to do with a git repository.

In my limited experience, packaging works very well with Flask: http://flask.pocoo.org/docs/patterns/distribute/

You just need a quickly whipped up Fabric script, and then you deploy remotely from the command line.

Although setuptools is capable of that kind of scenario, sometimes it's not worth the overhead.

A good rule of thumb is that if you have defined access points for others to consume your code with (command line tools, Python modules, or Setuptools entry points), it's a good idea to make a Python package like this. Otherwise, skip it.

I find setuptools works great for packaging whole apps with HTML/JS/CSS. You can put these data files in your Python package directory, list them in MANIFEST.in, and set 'include_package_data=True' in setup.py. Then you can reference them in your code relative to __file__. And you can include a run script for your app using 'scripts' in setup.py, or let setuptools generate it using 'entry_points'.

Take a look at https://github.com/getsentry/sentry for a good example. After doing 'pip install sentry', you can run the included gunicorn-based HTTP server via bin/sentry ('sentry init && sentry start'). A uWSGI-based server would have worked, too.

I like to create a virtualenv under /opt, install the package and its dependencies with pip, and then run fpm to create an RPM or DEB. Then I use Chef or Puppet to install the package, deploy config files, deploy a service script (for upstart or daemontools), and enable the service.

See also http://www.12factor.net/ for some principles on how to structure your app. These can be applied whether or not you use setuptools. For example, Heroku's Python buildpack will deploy your 12-factor app in-place using requirements.txt and Procfile in your app root, ignoring setup.py. I find 12-factor apps are easier to deploy whatever method you choose, since configs/services/logs are cleanly separated from your app itself.

There are a lot of interesting things in Senty. Its good reading though that code base.

I know this isn't exactly related to packaging the code, but any thoughts on utilities like py2exe (http://www.py2exe.org/), cx_Freeze (http://cx-freeze.sourceforge.net/) or pyinstaller (http://www.pyinstaller.org/) for distributing finished python modules to end-users?

You guys might also find this github project useful: https://github.com/splaice/py-bootstrap

ive been writing python for ages but never found a simple introduction to how to package modules - always thought it was voodoo, so thanks a lot! so simple...

It is but now you know the correct incantations ~computers~ :)

I apologize for the spam, but apparently I'm unable to save articles for some reason. I'd like to read this later, and I want to see if a comment will work.

If only there was some way to save a url within the software that we use to browse the web. That would be an incredibly useful feature, someone here should consider developing an addon or put in a feature request for such a thing.

Touche. Rather than ask other people if there exist issues with the intended story saving and voting features, I will start using browser bookmarks. Thank you for your valuable feedback. Pre-emtpive: "I know this is not the proper place to file a 'bug-report'"

*edit: not alone (semi-old) http://news.ycombinator.com/item?id=4164468

While I've already been doing things a similar way, it's nice to have a clean, clear explanation of how to package stuff.

Awesome job. I had been stressing about getting my code up there but your tutorial made it easy.

Thanks, this is exactly what I was looking for the last few weeks and couldn't find :)

Reply on news.ycombinator.com