Twisted History of Python Packaging | Hacker News Reader

Twisted History of Python Packaging

speakerdeck.com

38 pointsahammad15 comments

I'm a relative newcomer to python. I've been successfully using pip, and more recently pip+virtualenv to install packages. There haven't been many speedbumps so far, though I'm not hitting the system that hard. I do miss the "Gemfile" type of abstraction that Ruby or CocoaPods provides.

That said, the other day I tried to figure out how to package a bit of code I wanted to make reusable. It was a complete clusterfk. No two documents gave the same instructions, the instructions I did find were confusing and incomplete, and there was no obvious place to figure out what the "state of the art" of python packaging is.

I still want to package this module up, but it doesn't seem worth the effort so far.

well, since python is so much more readable than ruby you can just hand out a .py :P

I usually write python and just moved to ruby, and as much as I like python I don't think anyone brings DSLs into the conversation as much as it should be.

The whole advantage of packaging is `pip install mypackage`, which you don't get with a raw .py. and my module has a couple of non-.py resources contained.

I really really really really really really hate DSLs. That's part of why I like Python more than Ruby. I want to learn one consistent language, not dozens of magic keywords that apply only in certain situations. I recognize that there is a place for them, it's just that in my mind that place is called "Ruby". ;)

If you have a "simple" package, i.e., Python-only, and no namespace fanciness, find a setup.py from a package you use, copy it, scan and rewrite it, and you are probably good to go.

Just avoid:

- Namespace packages. This is where you have one package that provides, say, mypackageset.thing1, and another package that provides mypackageset.thing2. If you really want to have a "brand" with many packages in it, just name it mypackageset_thing1 and save yourself a headache.

- If you have important non-.py files (e.g., templates) in your package, you'll have to learn package_data (http://docs.python.org/2/distutils/setupscript.html#installi...) and maybe MANIFEST.in. These are annoying, and you won't notice when you install from a checkout or with "pip install -e". You might need to study up more on this thing in particular (or just poke around until it works).

- Some people use requirements.txt instead of install_requires inside setup.py now. I often use both – but with install_requires just for the absolute minimum of what might be required.

Here's the video of the talk (PyCon Canada 2012): https://www.youtube.com/watch?v=lpBaZKSODFA&feature=plcp

Thankyou.

I find it somewhat frustrating to follow a link that sounds interesting only to find a slide deck that makes little sense to me without the talk that goes with it.

In the Ruby world, a common heuristic for problems is:

1) Can the language do this for me? 2) If not, is there a gem that does it? 3) If not, can I make one?

The tools to make it easy to get and share gems and manage dependencies are a huge boost for all Rubyists.

Here's hoping that Python devs, who already have a very nice language to work with, can unite around an equally set of package tools.

Theres also Bento - http://cournape.github.com/Bento/html/index.html which is well designed. Disclaimer - David C is a friend.

I did not see in the presentation the following code snippet:

"from twisted.internet import protocol, reactor"

Somehow I feel cheated... ;) :P

Here is the video from PyCon Canada 2012 (with download link and such): http://pyvideo.org/video/1601/twisted-history-of-python-pack...

It's hard to tell with just the slides (and no talk), but it seems like this isn't presenting the complete/correct picture. Pip can use distribute or setuptools - it's not exactly an alternative as slide 31 makes it seem from just the text.

On a higher level, while it's interesting to know the evolution that things have taken, it's really not that complicated anymore - I haven't run into issues with Python packaging with ages; since pip is essentially fully featured (even including uninstallation), and pip is capable of working from source, I don't run into any issues when I use virtualenv (as everyone should be doing - and as is integrated into CPython since 3.3 anyway).

Python packaging has experimented with several different approaches, but it seems to have found the magical combination - in fact, I often miss the virtualenv + pip approach when working with other packaging systems in other languages (such as Haskell's cabal).

  >  it's really not that complicated anymore

Depends on your use-case. I wasn't at the talk (and haven't watched the video, yet), but apparently there was a bit of a 'heated' debate during the question portion of the talk. Apparently some people from the scientific Python community have given up waiting for their issues to be addressed and are working on their own packaging solution to meet their needs.

If most of your modules are pure-Python or maybe a little bit of C, then you might not be running into the warts that the current packaging systems have.

Also, running 'pip install package' hides what kind of lengths that developers/package maintainers have to go through to get things working correctly. That's also part of the discussion when talking about a decent packaging system.

  > I often miss the virtualenv + pip approach

Perl's "perlbrew + cpanm" approach is more fully featured. I miss having a 'requirements.txt' type capability, but perlbrew downloads/compiles Perl, so each environment can have a different version of Perl, which is a little more complex with virtualenv.

Python's not awesome packaging back in the easy_install days really put me off of using django as much as I could have. I liked python a lot, but I ended up in ruby more because gems were a much nicer way to go and bundler made it nice. Good to see python catching up a bit in that regard, but I enjoy ruby and Sinatra too much to switch back at this point.

So what exactly is the problem with easy_install? I just recently discovered it and it seems quite nice to me.

Package discovery was annoying. With Ruby, gems, and bundler I can define my project gems and install them with 'bundle install'. Any machine I move to it does that. It's fantastic.

With node and npm you can do more or less the same thing.

Maybe with easy_install you can now, but I remember having to manually download packages and easy_install them. No package management, no easy discovery, no project dependency management.

Reply on news.ycombinator.com