Explicit bootstrapping of pip in Python installations | Hacker News Reader

Explicit bootstrapping of pip in Python installations

python.org

298 pointsoellegaard77 comments

I kinda of like this... but then again I'm kind of wary.

Isn't the standard library the place where packages go to die?

Isn't the reason pip is actually useful because has a nice health release cycle (http://www.pip-installer.org/en/latest/news.html) and isn't frozen into the standard-library ice age of the past?

Won't this make it even harder to build a compliant version of python that runs on mobile devices where you can't install packages (>_> iOS)?

I get that it's convenient, I'm not 100% convinced its a good idea.

Edit: Reading the PEP in detail its now clear that this is not bundling pip with python (see 'Including pip directly in the standard library'). This is bundling a pip installer as part of the standard library. Much better.

Ya, this is a good thing. Ruby started shipping with rubygems since 1.9 and it's worked great.

And Node started shipping NPM in 0.6.3 --- which has also worked great.

Is it the "norm" that, when accepted into the standard library, packages "die" - or is it because this has happened to some packages, the popular perception is that the standard library is where packages go to die.

I'm just not understanding how it's OK that a standard acceptance of a Python package really means that it should go to the graveyard.

The standard library only gets new features when a minor or major Python release happens. That doesn't happen very often which slows down development of anything in the stdlib significantly to the point that packages in the stdlib effectively "die".

To some extent this is intentional because anything that requires more frequent changes is probably not stable enough for inclusion in the stdlib, in practice it does cause some unintentional issues for example rather lacking timezone support.

It looks like they're taking a new strategy with this move. Pip isn't actually going to be moved to the standard library to live; rather, a new "ensurepip" command that installs "pip" will be a part of Python proper.

However, a distributable version of pip will be included with each release so that ensurepip does not have to contact the PyPI servers during an install. And with each maintenance release of Python, this pre-included version will be updated to match the then-current pip release.

But, the bottom line is that pip still lives outside of the standard library. Python 3.4+ is just guaranteeing that you have a version installed.

According to the pep ensurepip was rejected: http://www.python.org/dev/peps/pep-0453/#including-ensurepip...

For backporting to 3.3/2.7. 3.4 has it.

> Won't this make it even harder to build a compliant version of python that runs on mobile devices where you can't install packages (>_> iOS)?

Why wouldn't you be able to install packages? iOS stops you from making executable pages, but cpython runs bytecode anyway.

Vaguely related, this wonderful article by Paul Tagliamonte on how pip coexists with distro package managers (in this case APT): http://notes.pault.ag/debian-python/

Nice reminder that pip is a dev tool and should be used as such. It makes sense to be included in Python.

I think that everywhere someone's instructed to use pip, it really should be to set up a virtualenv and then use pip inside that.

pip should not be going anywhere near your site-packages, and it'd be great to be able to disable that ability.

Glorious day! As a person who went to PyCon and experienced first-hand the state of Python packaging, this is excellent news. Good luck to the Python devs in the days to come.

Yes! and with pip you can uninstall packages while with easy_install you can't

Except for when uninstall doesn't work. Which is often.

I was always lucky then.

never happened to me in 4 years...

Yep! I haven't used easy_install in a long time, but that is true!

This isn't very big news, in that virtualenv already provides pip in each new env, and you should not be using pip outside virtualenv -- unless it's to install virtualenv!

We're not all building web apps in python - virtualenv is not universally useful.

(Edit: Not that I don't like virtualenv when it's appropriate, but it really bugs me the wrong way when people just throw out generalizations like that)

Having tried various ways of distributing desktop apps / non-web daemons, I've found virtualenvs useful there too (with the one fairly large caveat that installing wxWidgets in a virtualenv is a world of pain :( But then in those cases I use the OS-provided packages with the OS package manager, so I still never use virtualenv and pip independently)

> We're not all building web apps in python - virtualenv is not universally useful.

Huh, fair enough. I work on non-webapps too -- could you explain more about your use case?

Libraries that are very build-finicky (like various image processing libs) I've had bad luck with virtualenv, or env pip. Usually end up building by hand.

If you're doing data analysis, where most libraries are serious about backwards compatibility and you don't necessarily care whether your code still runs two years from now... virtualenvs are sometimes just not worth the bother. (Though I'd still recommend them.)

Exactly! I am currently working on a machine learning task. Python is great for this, but most of the scripts that I am writing are run-once scripts. VirtualEnv would be overkill of this usecase.

I'm not a web developer, but I do data analysis with Python, and I use virtualenv heavily. For one thing, it increases reproducibility to have a record (pip freeze) of which versions of each package I used.

I build web apps, but almost anything else I do, including a full Python based backend for $WORK is all done in a virtualenv. Virtualenv's are not just for web apps.

I agree with paulgb and others -- most of my past work with python could be called scientific programming in one way or another and pip has been enormously useful.

fully agreed. the numpy, scipy, matplotlib, PyQt4 stack in particular is very indifferent to virtualenvs: you're going to need to install a whole ton of dependencies system-wide regardless.

i recommend it to everyone i work with, but it is significantly more useful when all your dependencies are pure python.

when virtualenv was released with python 3.3 it didnt come with pip.

Didn't come with it, or didn't install it by default?

AFAIK the default virtualenv behaviour has been "create an environment and then automatically and immediately install pip from the internet", so for the last several years they've been practically tied together, even if they were distributed individually.

Didn't come with it, or didn't install it by default?

In that the following commands have also been required after running pyvenv-3.3, no, python has not "come with" pip:

  (venv) $ wget http://python-distribute.org/distribute_setup.py
  (venv) $ python distribute_setup.py
  (venv) $ easy_install pip

I believe it has been possible to configure pyvenv to do this automatically, but I've never done that.

I was talking about virtualenv, not pyvenv - grandparent saying "virtualenv was released with python 3.3" made it sound like virtualenv was released with python 3.3...

i think you are confusing virtualenv with pyvenv which is what came with python 3.3 http://www.python.org/dev/peps/pep-0405/

I have no idea how to get virtualenv working. I'm that user that needs it for now!

Install virtualenv:

    pip install virtualenv
    mkdir ~/.virtualenvs
    virtualenv ~/.virtualenvs/my_new_project

Activate it to continue development:

    . ~/.virtualenvs/my_new_project/bin/activate

The effect of this is that `python` and `pip` commands now only act on the virtualenv and not system (or user) site wide.

Deactivate it:

    deactivate

Python and pip commands are now the system versions again.

Craig, you're my internet hero today. Much thanks!

virtualenvwrapper will be a life saver for you.

Either way at some point you need to install pip. They are removing an unnecessary step, which is a good thing.

I'm not really a pythonist so I'm not 100% aware of the consequences of this, but as someone who deploys Python-based software every now and then, this just seems to make sense to me.

About time really.

As indicated in the Tagliamonte piece linked above, be careful deploying with pip. OS or distro tools such as apt-get are usually more appropriate for deployment. Distro maintainers have made commitments that PyPI uploaders have not made. Of course sometimes users need things that are not available from a distro repo, but in that case they're not so much "users" as "testers".

I really really despise what distro maintainers do with packages. Pinning some stuff way in the past, esp deps used by lots of projects, this then forces consumers of those deps to get patches against their code to work causing a schizm in the public version and the distro version of the library. A huge pointless wasted mess.

I almost always use pip and my own installed code (kinda homebrew like) to put dependencies onto a box. It takes more work but I can skip the bullshit.

Yeah I get that. If it's a frequently updated package that you use frequently and care about, or on an instance you're both developing and maintaining, it's totally worth it to do the system integration for yourself. I got the impression that GGP was deploying for other people. In that situation, not having a distro package maintainer for a package might mean the sysadmin becomes a de facto package maintainer, which sucks for the sysadmin and the users both.

For my own education: Python with a good standard packaging system and solid, standard async capabilities would be solidly going after the same areas Node.js has done so well in? If not, why not?

I doubt it. All Node.js libraries are built around the non-blocking model which is why it works so well. Python libraries will have to decide if they're a normal blocking library or an async one. Possibly maintain both versions of the API?

Personally I'm not convinced that the Node.js callback hell is a good way to get high IO performance. The Golang way of doing it with normal, blocking libraries and Goroutine's seems ideal as it doesn't burden the programmer as much.

Callback hell isn't, but promises do help (to a point)

In 0.12, node will ship with a version of v8 which has generators (hidden under the --harmony flag), which are shallow coroutines, which allow for the same code style as go-routines.

Existing non-blocking libraries will be reusable with generators via e.g. https://github.com/jmar777/suspend or https://github.com/visionmedia/co etc.

Performance could also potentially improve as in V8 closures are much more expensive than generators.

Good to hear. The situation with python packaging has seemed kind of chaotic for awhile. The setuptools/distribute merger will hopefully standardize things from here on out.

I like pip, but it's too bad that it can only install from source. It's quite a hassle sometimes to round up dependencies and build them all on windows (not to mention not everyone has a compiler installed on windows).

easy_install can install from binary installers or eggs. I'd like to see that added to pip.

Not being able to install from binary installers is so annoying, it's a real shame it doesn't support it. Its the only reason I keep easy_install around.

This is blatantly untrue, pip installs wheels. eggs are dead, easy_install is dead. let them die. We are talking about Python 3.4 here, after all.

I don't know much about the wheel format. But eggs being dead is news to me. You have links for those interested in more info?

here are some wheel links:

https://pypi.python.org/pypi/wheel

https://wheel.readthedocs.org/en/latest/index.html

https://bitbucket.org/dholth/wheel/

http://www.python.org/dev/peps/pep-0427/

i am using wheel (with pip) as a replacement for .exe installers to install some packages (numpy, scipy, py2exe, etc) on windows. i find it useful because you can automate the installation of a wheel archive using pip (with the typical .exe windows installers you need to click next several times...). the wheel command line utility also knows how to convert existing `.exe` installers to wheel archives.

I read all that, I was more interested in information on the "eggs are dead" assertion.

That feeling of having to keep GBs of Visual Studio 2008 installed merely to be use the most recommended python XML library (lxml). Alternatively you can just add a 2MB egg to the repo.

What is involved in getting a wheel file to pypi so that `pip install lxml` happens out of the box in windows?

Nifty, always nice to save a little time on future installs

More importantly, it's nice to assume your users will have pip available no matter what (assuming we all move to Python 3. One day...)

Python 3.4 seems it will be true to it's promise. See the other thread: https://news.ycombinator.com/item?id=6581053

As someone maintaining packages on pypi, I can tell you that demand for Python 3 support is growing and people are beginning to port packages for their own needs.

I expect we'll see python 3 overtake python 2 in new projects within 3 years. I realize that's still pretty far off, but these things take time. You have to give the PSF credit for great support of the 2.x series.

As a recent convert to Python (but only as an enthusiast... still doing C++ in my dayjob), I chose to go all in and start out with Python 3. I'm glad I did, but I do need to keep version 2 around.

Nice... although I don't get why pip is so much behind RubyGems and NPM in terms of package management. pip should merge with virtualenv and virtualenvwrapper as well.

Python packaging has been a nightmare compared to RubyGems (which is not without its own problems, of course). See for example this rant by a major Python developer: http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/

This sounds like a great step forward to me in making Python packages easy to create, install, and uninstall!

Bundling the subpar pip actually solidifies the status quo and is in no way improving the situation. I think what made both Ruby and Node.js great is the flexible package management.

pip continues to improve and they are providing it outside the stdlib.

They surely do... but at what pace?

Hopefully it will also bring about some improvements to pip. It's a pretty great tool but with a couple major caveats. The first is that although it supports many forms of package specification, including VCS repositories, it does not report the package spec according to the way that it was installed, but rather according to the package name and version according to its setup.py. Say you install a package from a git commit that fixed a bug in the PyPI 1.0.0 package whose version is still reported as 1.0.0 in setup.py at the commit. Then you freeze the environment to requirements.txt to distribute. It's still reported as package==1.0.0 instead of the git spec, so the next person to install will pull down the broken version from PyPI instead. The other headache is that installing from a requirements file just installs dependencies in the order they're listed, so oftentimes you need to re-organize the output of `pip freeze` to make sure dependencies are installed in the right order, otherwise you can encounter things like unexpected package versions due to other packages making ambiguous dependency specs for dependencies of your own app.

I have to say, these are kind of obscure corner cases. Should be fixed, to be sure, but I don't think they are an accurate representation of pip to people who are not familiar.

What do you expect me to use as an alternative? easy_install?

You're right, maybe I've mis-categorized as them major caveats, but they're not so obscure because both myself and others have had to wrestle with these properties in production. I'm not saying use easy_install by any stretch (pip is great), they already have open tickets and I hope to see them fixed before being rolled into Python proper and do become defining problems within pip.

Yey!

Look at the trails Node + NPM have blazed. That's the right move.

Next up: Local package installs.

I don't think effective package management solutions were really pioneered by node and NPM.

You could pretty easily re-write your comment and say "Look at the trails Debian + apt have blazed. That's the right move."

Or, for a more apt (heh) comparison:

https://github.com/rubygems/rubygems

"If you are writing a package manager for a new non-js language, do whatever it takes to get that new language to load modules the way node does. The right way."

http://dominictarr.com/post/25516279897/why-you-should-never...

http://dontkry.com/posts/code/modules-the-right-way.html

To be more specific, what npm does that pip, debian, and rubygems don't do is nested dependencies, which is a wonderful advancement.

Isn't virtualenv going to be bundled too?

a flavor of virtualenv was bundled in 3.3 (http://www.python.org/download/releases/3.3.0/) but without pip or easy_install it was a pretty useless feature IMO.

rubygems has been bundled with ruby since 2009, and perl has had cpan since about forever.

Python has had a good installer forever. The only difference is that it wasn't centrally mandated. If you need something to be centrally mandated in order to use it, I feel for you.

It does help to know that it exists.

npm badly re-implements what others have done dozens of times. The whole node ecology suffers heavily from "not invented here" syndrome.

Reply on news.ycombinator.com