Python packaging is good now
glyph.twistedmatrix.com
glyph.twistedmatrix.com
Don't get me wrong, I love Python and I'm not so much a fan of NodeJS, as Python is so much better designed than JavaScript, even in time of ES6. But their NPM is really well designed. NPM alone gives NodeJS a competitve advantage over many other popular languages including Ruby (Gem) and Python (PIP).
For example, why does Python packaging need such a complicated bootstrapping process? Install ensure_pip, then the real pip, and on top of that virtualenv. NodeJS just ships with NPM. Done.
Any why do we need to create and manage a "virtualenv"? Why do you have to go through extra hassle just to avoid global installs? NPM just creates a local sub directory node_modules and automatically installs and removes from that. Global installs happen only if you really want it to (npm -g). But you usually don't.
The only central place at NPM is the package cache so it doesn't download things twice if you have two separate projects with similar dependencies. In the Python/PIP world, everything is central and even today you still have to accept extra hassle to get a sane, local setup.
Other than that, I agree.
> "For example, why does Python packaging need such a complicated bootstrapping process? Install ensure_pip, then the real pip, and on top of that virtualenv."
It's not that complicated:
sudo apt-get install python-pip python-dev build-essential
sudo pip install --upgrade pip
pip install PACKAGE
>"In the Python/PIP world, everything is central and even today you still have to accept extra hassle to get a sane, local setup."
I think you may be coming at it from a javascript perspective where you expect it to work according to the way you're used to it working with Node/NPM. I have absolutely no hassle installing anything in the python/pip ecosystem. Except maybe MySQL and OpenCV, I'll grant you that. But those are separate discussions as they require external dependencies outside of python's packaging.
The main issue is that they're modified at all, so they don't match up with the docs or what people say online, so until you realise that you're not running the normal code and need to run inside a virtualenv it's very confusing.
> The main issue is that they're modified at all
Not modifying packages at all would also be a major issue. We'd end up shipping dysfunctional packages, or packages with major bugs, or not fixing bugs during the lifetime of a release.
Distributions do a ton of integration work that I think goes unnoticed by users. We routinely find issues and send them upstream - every development cycle. Users don't notice because they'll also find the bugfixes in upstream releases because we sent them there. But upstream releases don't all happen in lock-step, so we end up having to carry some patches to make everything work with everything else.
In the general case, we can't win. We'll always upset someone, which is why I asked for specifics. In general, we will:
(0) Avoid changing anything in a stable release, so users don't have things changed on them - except for (1).
(1) Fix bugs (including security bugs) as they are reported. Because of (0), this generally means that we cherry-pick fixes, which does cause some divergence from upstream. But if we don't do this, then other users complain either about the bugs or, if we pull everything into a package in an existing stable release, about things being changed under them.
(2) During development of a release we fix for integration issues, since every package in the distribution is expected to work with every other package - and (though with some exceptions) we generally only ship one version of each thing in a release. This means that we sometimes need to patch things (we prefer cherry-picks from upstream, but sometimes we don't get a response from upstream in time for our release schedule) in order to make everything work.
But in the specific case of PIL, we appear to not have had to do any of these things, except for some security fixes.
For urllib3, it looks like we had to make some modifications to fix security issues with regards to TLS and certificate verification in Trusty, and one bug in Xenial related to broken IPv6 square bracket handling, and that's it.
I don't really see how we can do anything different here. Should we have ignored the security issues and shipped vulnerable packages? Or should we have upset many more users of the stable release by bumping behaviour under the feet of existing deployments?
The maintainer approach works well for everything you maintain, it just sucks for everything else, including installing out of repo software or writing your own software. It's made particularly bad by Python's packaging system which defaults to global installs.
FWIW, I lay this one more at the door of Python than I do Ubuntu, you guys are just dealing with their packaging system and I think that "Python packaging is a good now" is just ignoring the shitty parts.
Frankly it's easier for me to just always run my code inside a virtualenv than it is for you guys to make some drastic changes to make my life marginally easier, when I don't even really know what the problem was. It's just that this is a non-obvious thing to do for someone who hasn't been in the python ecosystem for long.
If this is the case here, then this isn't a Python-specific problem at all. Developers have a choice: use the chosen distribution release version, or if that's not new enough, then find another way. For Python, one (commonly accepted and good) answer happens to be virtualenv. Many choose, because of their particular circumstances, to go all virtualenv and ignore distribution packages altogether.
Can I guess that you used an Ubuntu LTS, so you effectively opted in to packages that are older but don't change often?
Some solve this by never using a distribution version at all, which is fine. But understand that this is the nature of what a distribution is, and applies to the entire ecosystem, not just the Python one.
Do I have this problem in Java/Node/Go/Rust? No, I don't.
The reason I don't have this problem is that dependencies are not installed globally, they are defined per-project and often statically linked in, there is no such thing as a conflict between two of those projects.
It sounds to me that you hit some kind of straightforward bug then, rather than some systemic issue.
Yes, eventually you'll (hopefully) upgrade all your projects to the newest and shiniest versions of all your dependencies, but if you need to maintain some semblance of stability, that's not always immediately possible/practical.
But that's not the point. The point is, if I'm on a system where I don't have the ability to make system-wide changes (run docker as root, add myself to a special group, etc), then I can't run docker on that system.
(Invoke-WebRequest -UseBasicParsing "https://bootstrap.pypa.io/get-pip.py").Content | python -
but in most cases, the `curl url | python -` or equivalent solves the problem
Please avoid making such assumptions about other people. It's just a little step from there to ad hominems.
To make my point more clear:
I love Python and 'm used to Python/PIP, but when I had to work with NodeJS and saw NPM, I immediately thought: These people got it right. Not the language, mind you, but the package management!
I don't really see the issue with creating a virtual environment in the case of specific dependencies, and otherwise dealing directly with the global environment (which really isn't that different from e.g. NPM and JS, when you think about it).
What's a bother is - as that article mentions - end-user deployments.
Now, I sort of have that down to a tee by using cx_freeze and Inno Setup, such that I can distribute my software to my end-users and all they have to do is click on a Setup.exe and my applications install just like any other Windows program.
I can do a similar thing on the Mac, but it's still a pain.
The main trouble with packaging Python applications up, is an apparent assumption (at least, that's the impression I have gotten over years of figuring out how to distribute my applications), that Python programs should all be open source - so for example, distutils assumes that the author wants his code to be readable by everyone who wishes to.
I wanted to be able to produce full applications and somehow compile and package them as in for e example, a C/C++ application.
The closest I can get to that currently, is cx_freeze, which isn't compilation, merely turning the Python code into bytecode.
Nuitka is showing lots of promise, but as yet it still cannot compile my applications into something usable (my really big Qt4 GUI applications either won't compile or there are lots of problems cropping up even if they do manage to run).
Here's hoping end-user deployment gets much easier than it is now, someday!
1) Develop your Python application ;)
2) I have a batch file build.bat which I use to build the deployment folder. This sits in the root directory of every application I develop, and is customised. It basically calls cx_freeze via the command line rather than bother with a setup.py file - I find this method far quicker and more convenient. An example...
cxfreeze -OO -c CaptainsLog.py --include-modules lxml._elementpath,lxml.objectify,win32com.gen_py,requests.certs --base-name=Win32GUI --target-dir ..\CaptainsLog_DIST --icon CL_new_256.ico
cx_freeze will build and include all the necessary Python and Qt libraries to ..\CaptainsLog_Dist
I've had to ensure cx_freeze included some required stuff like lxml and those other things because for various reasons it didn't automatically include them when they were not specified, so be aware of that possibility when attempting to freeze your application.
3) I then can test whether the build/distribution works by double-clicking on the captainslog.exe file which cx_freeze has produced.
4) If all is well, I then run Inno Setup. I have a file captainslog.iss which I use and change as required (e.g. new version number). This will basically take everything which is in the CaptainsLog_DIST folder, and package it up into a suitably named Setup.exe file.
5) I then sign the Setup.exe file with my code signing certificate. Search google for how to obtain such - basically you need at least a Class II certificate from a suitable provider to prove your identity. I use a utility called ksign to do the actual signing.
6) I test the Setup.
7) All going well, deployment to my users is simply providing a download location - notification is built into my application (check for new versions), or if they disable that, peruse the game forums or my web site.
It's all been rather streamlined over a LOT of trial-and-error, frustration, and sometimes a lot of swearing. But the above works REALLY well for me and I use this workflow for bespoke software I write for paying customers.
It's a similar workflow (with some differences) for Macs.
The real beauty of it is that your end-users don't have to worry about installing Python, Qt, any of the Python dependencies/packages - everything required to run the application is provided.
The other bonus is that you only need to worry about Python dependencies at /your/ end.
Note: when developing and deploying my applications, my entire toolset is the 32-bit versions - this ensures that your package will install and run on both older 32-bit and newer 64-bit versions of your target O/S.
Regards.
Is there a problem with that?
If you're worried about someone stealing the code you can obfuscate it a bit.
Yes - I wish I could compile my applications. Certain ones could benefit from being sped up a lot.
I have used Cython before, on a pure Python program and using the --embed command, but it was a LOT of work to get that working (ended up having to take my Python program and merge everything into one monolithic Python file and feed that to Cython. And that meant taking the output from pyside-uic and bunging that in, as well as merging in the Resource file generated from pyside-rcc.
I did that as an exercise. It worked, but was an awful lot of hard work and it was a horrifically hacky way to do it ;)
At the same time I did also obfuscate things like the Licensing code that was in that file, in Python, beforehand.
So yeah, it's not like I'm new to that sort of thing :)
I just wish there was a more effortless way to do that - Nuitka has that promise but last time I checked it's just not there yet, despite the awesome work being done by Kay.
I created something similar to that here [1] for an abandoned project (but the project is not about self-extracting archives, so don't bother compiling the project).
(1) https://github.com/RamchandraApte/unvanquished-installer/blo...
Is there any reason not to write your app in Cython? Then you could compile it with your ordinary C compiler. Could be much harder to reverse engineer that way, if that's your concern.
My reply at https://news.ycombinator.com/item?id=12286806 is relevant.
Regards
My intention was to create a Python 3 package using best-practices, a nice division of labour by process to avoid threading problems, etc.
So far I've run into the following issues regarding packaging:
Getting started - there's a glut of documentation, much of it contradictory, much of it out-of-date. Simply google "Python packaging" and you'll have to pick between 4 sources of info on the first page, all of which look official.
The tools - this one was a new one for me; I normally use virtualenvs for development, but vaguely remembered it being bundled in Python 3. Turns out it's not virtualenv but "pyvenv". Beyond the confusion of the name change, there's also a tool called "pyenv" muddying the waters.
The specifics - I started with https://packaging.python.org/ and was struck by how ambiguous the instructions were. Doing something as simple as declaring your package's version has 7 different options: https://packaging.python.org/single_source_version/. In the end, I got the most help from reading the guide's sample project... I only have to wonder about the licensing implications of that.
The infrastructure - because I was creating a project with multiple executables, I needed a way to start one from the other. Despite knowing that setuptools is responsible for generating the executables, I quickly found that there's just no way to find those executable paths programmatically.
Library choice - I needed an inotify library. I searched for "inotify", "notify" and "watch" on Pypi but had little way of distinguishing the results - the "weight" column is useless as compared to a rating system or even download count. Thankfully, I remembered about awesome-python.com which bridges the gap.
All told, it took me several hours to bootstrap the project, not even including things like testing. I'd say Python packaging has a way to go before I'd call it "good".
I then use `nosetests -v` to run the tests.
[0] - github.com/ambitioninc/ambition-python-template
On Debian there is Debian Python, and Debian pip. Then there is easy_install to install own pip, and that's different. virtualenv comes on top of all. Of course, there is Debian's Python packages, too.
On MacOS X with Macports, there is Apple's Python and Macports' Pythons, all with different pips. Of course, there is Macports' Python packages, too, for different Pythons.
The primary reason is that setup.py, etc., have you download/manage one set of dependencies for each project you're working on rather than simply housing all dependencies in a common folder, separated by version, as Maven does. It's pointlessly redundant. But the real pain comes when you start to install things. Since in Python land, dependencies for all tools are typically stored in the same folder, you very easily and quickly hit version conflicts since the dependencies are not really segregated by version. This leads to brutal bugs that creep on you at runtime when you realize, too late, that some tool you installed has dependencies that conflict with another.
What's worse, complex Python projects like OpenStack may have numerous dependencies which all in turn have differing and conflicting dependency versions of their own. Trying to keep them all in sync is a ridiculous exercise in time wasting and pointlessness, and problems caused by version mismatches are common.
The brilliant solution (sarcasm here), of course, is to create virtual environments with separate copies of every dependency for every tool we run. Again, pointlessly redundant and a pain to manage.
Is there a better way? Obviously. Use a single common repo with dependencies structurally segregated by version (ie: different folders), as is the way of doing things in Java-land. I know that Java-land stuff is bad and we're all supposed to hate it. But I've always failed to understand why developers in other language ecosystems continue to create tools in a vacuum, as if roadmaps for solving these problems don't already exist elsewhere. Look beyond your bubble. Hopefully one day the Python folks will catch on.
> The brilliant solution (sarcasm here), of course, is to create virtual environments with separate copies of every dependency for every tool we run.
Brilliant solution is to recognize that environments need to be packaged and managed too. No sarcasm. That's what nix does. And it makes package management a solved problem (ignoring current implementation-specific issues of course).
So, yes, there is a better way and it's nix [1] and guix [2] and the ideas behind them. Check them out.
The real takeaway here though, is that although it’s still not
perfect, other languages are no longer doing appreciably
better.
At least not true in the context of OCaml/OPAM.
As always, I’m sure none of this applies to Rust and Cargo is
basically perfect, but that doesn’t matter, because nobody
reading this is actually using Rust.
https://github.com/servo/servo
https://github.com/dropbox/rust-brotli
--
Bottom line: Bash everyone else arbitrarily and claim oneself to be the best!
Wheels suck.
By using the bdist mechanism they make it impossible to build a single cross-platform, cross-Python package that just works. Wheels can't run 2to3 at installation so guess what? You have to make separate packages for Python 2 and 3.
There should be an easy way to have a pure Python package that "just works" on Python 2 and 3, 32-bit and 64-bit platforms without having to build separate packages. Right now that can only be done with the third-party Setuptools using sdist and a traditional setup.py. Distutils is still a half implemented packaging system. I wouldn't say that Python packaging is a solved thing.
wheels are unbelievably critical in that they allow super easy caching of pre-built packages on any given host. If you've noticed these days you can type "pip install numpy" at will and it seems to usually run in less than two seconds rather than 5 minutes, thank wheels. This is particularly a big deal if you work lot with CI.
That's a strange thing to say. Six and future were written to ease the migration to Python 3, and both are endorsed by the official Python porting guide (https://docs.python.org/3/howto/pyporting.html). Meanwhile 2to3 is problematic because it complicates debugging and doesn't address issues like string handling, backports and other behavioral differences. Few popular Python libraries use 2to3, and most embrace the idea of single-codebase cross-version support.
I would agree that the Python core developers bungled the migration tools in the standard library, and I wish they were better. But as noted by the OP, library functionality in general is steadily becoming more decentralized, and six and future are the best options for packaging cross-version libraries today.
More generally, a lot of good work has gone into the wheels architecture, and it's a huge improvement on what we had before. If you think you can come up with something better, I would encourage you to try - the opportunity is there.
Some things have improved, but fundamentally there really is not much difference in creating a package now vs. then.
A certain xkcd comes to mind.
There are some out there, e.g. OpenStack uses https://pypi.python.org/pypi/pbr - it was pretty painful around the 1.0 transition when no two libraries seemed to be using compatible versions, but it hasn't caused me any trouble in quite a while now.
I'm more concerned by the lack of a (non-greedy) dependency solver in pip.
https://github.com/pypa/pip/issues/988
''Pip needs a dependency resolver''
Without such a thing you really don't quite have a way to ensure what you are requesting to be installed actually is a valid set of things.
It creates a lot of work to align your dependencies.
It's bad enough that I have a boilerplate `setup.py` that just does `import ConfigParser` and blasts the entire `[setup.py]` section into `setuptools.setup()` as `kwargs`. I copy and paste this boilerplate into every new project that I start.
Oh and while we're ranting about Python packaging, let me rant about tox, which is so-very-useful but also so-very-dangerous: IF YOU (SLIGHTLY) MISCONFIGURE YOUR TOX.INI FILE, IT WILL DELETE YOUR ENTIRE PROJECT DIRECTORY! [1]
I suppose I am getting too old and cranky for this stuff. :)
[1] https://bitbucket.org/hpk42/tox/issues/352/tox-deleted-all-o...
If you're not prepared for that situation at all times, you are using computers wrong.
[1] https://www.python.org/dev/peps/pep-0518/ [2] https://github.com/pypa/pip/issues/3691
Python Core started distributing the ensurepip
module along with both Python 2.7 and 3.3 ...
$ python -m ensurepip --user
Turns out according to one of the links given by the article[1] that this feature was actually not backported to these older versions of Python, apparently to encourage people to upgrade to new Python. This is unfortunate.
[1] https://www.python.org/dev/peps/pep-0453/#including-ensurepi...
$ python2 -m ensurepip --user
ensurepip is disabled in Debian/Ubuntu for the system python.
Python modules For the system python are usually handled by dpkg and apt-get.
apt-get install python-<module name>
Install the python-pip package to use pip itself. Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.However, now you have me worried that the article left out caveats and warnings about how these commands might lead to "unexpected results".
1. Install system python
2. Install pip from somewhere
3. pip install requests
4. apt-get install something with depends on python-requests
Should the installation of apt's requests fail? Should it work but clobber your pip installed requests? What happens when the apt version is older than the pip version? Does this break whatever you used pip install requests for?
There's two resolutions to this issue.
1. Don't use pip, always use apt for your python dependencies. The Ubuntu devs would really prefer you pick this option but not everything on pypi is on their repo and sometimes working consistently across platforms is more about than within platform consistency. 2. Use a different python environment than the system python. This could be done by using virtualenv, a different python version or a separate installation.
This was Ubuntu bug #1290847, which is now marked as "Fix Released": they consider the current behavior okay.
Python packaging is good now, except on Ubuntu.
Ubuntu is hardly on its own here. I recommend that anyone who disagrees with how Python is packaged on Debian and Ubuntu get in touch on the relevant mailing lists. If you think it's wrong, there's probably a use case you're missing, or you'll find that your proposal will badly break some other expected behaviour.
$ python -m ensurepip --user
$ python -m pip install --user --upgrade pip
$ python -m pip install --user --upgrade virtualenv
> From here on out, now the world is your oyster; you can pip install to your heart’s content, and you probably won’t even need to compile any C for most packages. These instructions don’t depend on Python version, either: as long as it’s up-to-date, the same steps work on Python 2, Python 3,
Nit: I suppose installing virtualenv works in 2 and 3, but hasn't venv in 3 made that unnecessary?
For example, with virtualenv you can choose to use a mix of globally- and locally-installed packages. With venv you can't. (You get global or local, but not both.)
The database driver and scipy are installed globally. Everything else is local to the environment.
Not hell at all, especially compared to compiling numpy/scipy every time you want to spin up a venv that uses a single feature.
Examples other than DB drivers and scipy exist. Compiling software isn't really that hard, but it's an unnecessary hurdle in many cases - especially when you've already bought into an OS that does binary distributions specifically so you don't have to compile anything.
I'd be comfortable agreeing that any pure-python package should be venv only, but not that every package should be.
I find the griping a bit over the top. The point of the OP is that new exciting improvements have arrived in the Python packaging infrastructure. Yes, there were many ugly overwrought things in setuptools' past. Yes, npm has several great ideas. npm is also much younger than setuptools/PyPI, and I can cite lots of horror stories from its pre-io.js days too. But the volunteers who got us this far deserve to be recognized for the progress they've made.
I think the biggest legitimate concern right now is how unintuitive it is to use pip in the presence of several Python runtimes/versions.
But for a beginner who is just trying to figure out how to share his or her code, it's still very confusing. And yes, part of that is definitely the fact that all of the mistakes of the past and their messy workarounds still show up in Google searches, muddying the waters; this obviously gives npm an edge. But even so, I feel that npm just does a better job when it comes to being learnable: people pick it up right away with very little trouble, and you can have your first "public-facing" module up on day one with very little effort. Indeed, in a way, the fact that it's so easy is what's giving it the most grief right now!
So there are definitely lessons to be learned there, for everyone.
Everything else is pretty much obsolete.
The problem is in how it doesn't really work reliably, especially with C extensions, which are much more common in Python-Land than with anything Javascript/NodeJS-related. You're also comparing with something that, by virtue of its age, has far less technical debt. Also, NPM is a "commercial product", PyPI is a community effort. Maybe the comparison to something like Anaconda would be more apt.
The author acknowledges most problems anyway, he just comes to the wrong conclusion: It's not "good" now, it's still terrible, but everything else is more-or-less as bad or worse.
Yes, pip, since before bundler existed.
> Suppose for example that I want to upgrade a package and all packages that it depends on, how would that be done?
pip install --upgrade package
> suppose I have different environments a project can be run in that should load slightly different set of dependencies?
venv activate myproject
I've run into bugs where someone ripped a library out of requirements.txt because they thought it wasn't being used any longer. Turns out it was still being used, but the dev couldn't tell because the lib was still in their venv. Tools like Bundler will actually take the dependency away from you.
Python's packaging system is great compared to C, but it's primitive compared to Ruby, Haskell, Clojure, and lots of others.
Maven and co.
> Suppose for example that I want to upgrade a package and all packages that it depends on, how would that be done?
Change the version number in pom.xml/build.sbt/build.gradle. Done.
> Or suppose I have different environments a project can be run in that should load slightly different set of dependencies?
Define a new project that depends on your main project and the extra dependencies.
"How do I make an exe" with my PyQT app is still a very hard question to answer.
Web deployment is still hard in Python because of lack of general hosting support if you don't own your server.
The situation is way better, and I'm very glad of the work done, but it has still a long way to go.
Does conventional shared hosting actually matter anymore, for application developers as opposed to someone just running a WordPress site? Virtual private servers are just as cheap now.
GAE? PythonAnywhere? Heroku? OpenShift? Docker-based stuff? https://wiki.python.org/moin/SpecializedCommercialHosts maybe? https://wiki.python.org/moin/OtherCommercialHosts if that wasn't enough?
An even simpler way to think about it - imagine you just invented python and are designing the tooling around dependency management and packaging. Would you come up with 'environments'?
What? No, they don't. Typical Java apps are deployed on systems that don't even have Maven installed. Typical Java apps are executed with a classpath argument pointing to the libraries installed for the application. What are the directories containing those JARs (or an uberjar) except an "environment?"
Hell, the Java classloader interacts with the classpath in pretty much the exact same way that the Python interpreter interacts with a virtualenv.
Compare to a jvm uberjar.
...but, the deployment is a single artifact.
I don't think doing a git clone, followed by a pip install and waiting for all the dependencies to download... and compile... (or not, if you have the wrong system packages installed) really compares.
If you are doing a blue/green type deploy.. you are basically building up a new VM to be your new environment. It can do all kinds of download and compile type stuff. Who cares if some is pip installing?
a) A 30 sec - 10 minute process to build (vs 0 in python)
b) Weird because Java has a fatjar concept.
Python and Java are different. The easy path to deploy them doesn't need to be the same.
You may need to make them the same for ops sanity.. but if you are just doing 1 or the other, meh
Have you ever run someone else's python code? You have the exact same issue pulling down their requirements.txt, which is often not enough because you need to apt-get other libraries, try pip again, apt-get more, try pip again, etc.
> b) Weird because Java has a fatjar concept.
Sure, that's what you would do if you wanted to deploy an application to proc, my point was that it supports the same git checkout workflow if you want it to, which can be useful when dealing with other people's code and you want to make changes and not just run whatever jar they made.
Zip it all up and then deploy with `pip install --no-index --find-links=/path/to/wheels -r pinned-requirements.txt`. Et voila! Binary artifacts that don't suddenly break because PyPI goes down or a package maintainer uploads a new version that unexpectedly deprecates an API that you're using.
So honest question, what's this easy_install stuff? Apt has been around since the nineties, just like python, and didn't change much as far as I know.
I also don't use Debian.
Far from it. You get repeatable builds and deployments, much easier and faster setup of development environment (GCC and -devel packages and foreign-language tools pulled in automatically), and you don't make intractable mess on your servers, which allows you to use much wider range of tools than architecturally broken Ansible/Salt.
> I also don't use Debian.
Irrelevant. Every Linux distribution and most other unix operating systems use some package manager, most of which in turn are more or less equivalent to APT.
easy_install [0] installs remote packages (similar to CPAN for perl). Obligatory reference: Why use pip over easy_install? [1]
[0] https://wiki.python.org/moin/EasyInstall
[1] http://stackoverflow.com/questions/3220404/why-use-pip-over-...
1) I need to have two applications running on the same host with different Python versions. E.g. Python 2.6 and Python 2.7. apt would only support one of the Python 2.x line installed at a time.
2) I need to have two applications that have conflicting version requirements on the same host. E.g. neither set of requirements can be installed at the same time.
3) I need to install a newer version of a Python library than the one in apt. If I install it into the system Python via pip, now apt is out of sync with what is actually installed.
4) I need to install a version of a Python library that conflicts with the requirements of system tools that rely on the system-installed Python. If I install said library, then the system tools will stop working / break down.
Also "checkout into your project directory and call `import`" doesn't scale. Maybe it works well for toy projects and one-off things, but not so much at scale.
I take it you haven't used Debian/APT for last three or four releases, because what you assert (only one Python 2.x version per host) is just plain invalid.
> 3) I need to install a newer version of a Python library than the one in apt. If I install it into the system Python via pip, now apt is out of sync with what is actually installed.
Uhm... I don't know why you have a problem with building your own package with the newer version. I do it all the time, for software both others' and mine.
This leaves points 2) and 4), which are essentially the same: how to install two conflicting versions of the same module for two different applications (one of which may come from the distribution), but this is not a problem with dpkg or RPM, this is a problem with Python's module lookup mechanism. For C libraries there are already good protocols for handling this issue (a little different for RPM and dpkg).
Maybe things have improved in the past few years, but last time that I built a dpkg for a Python library it was far from painless, even with some wrapper off the Debian wiki specifically for that use-case (and I was just doing this for pure-Python).
I've dealt with building both RPM and DPKG packages in the past, and I recall all of the build tools for each being far from ideal, and even the use case of "I want to package up a list of files and install them to a specific location" was far more difficult that it should have been. The biggest issue being that most of the tooling is based on the use-case of taking a tar of sources files, and running it though autoconf-esque build process to build it, apply patches, etc, which might be great for prolific package maintainers, but make things a bit more daunting for the new / causal user that just wants to create a simple package.
> how to install two conflicting versions of the same module for two different applications (one of which may come from the distribution), but this is not a problem with dpkg or RPM, this is a problem with Python's module lookup mechanism
Regardless it is an issue that exists. The parent post was saying that "apt should be good enough for all use-cases" which I was rebutting. I wasn't claiming that APT was sucky / broken, but that it doesn't handle all Python use-cases well. Whether it is the fault of APT or Python as to why they might be incompatible in a specific use-case is sort of irrelevant if you end up needing account for said use-case.
> For C libraries there are already good protocols for handling this issue (a little different for RPM and dpkg).
I'm not fully versed in the mechanisms in C, but IIRC, isn't that why packages are renamed when breaking changes happen over major versions (e.g. libxml vs. libxml2 / libgtk2 vs. libgtk3)? "Rename the package" seems more like a work-around than a system to handle the issue at hand. You're not saying, "I want libgtk with version 3.x".
> Maybe things have improved in the past few years, but last time that I built a dpkg for a Python library it was far from painless
"Past few Debian releases", since debhelper 7.x, not "years" (well, this incidentally matches, too). Debhelper can build properly Python using setup.py and doesn't require anything special (no need to specify any paths) to put artifacts where they can be found by /usr/bin/python. It's actually easier than building analogous C code, since you don't even specify where to put what.
> [libxml vs. libxml2 / libgtk2 vs. libgtk3] "Rename the package" seems more like a work-around than a system to handle the issue at hand. You're not saying, "I want libgtk with version 3.x".
For DEBs, yes, this is the case, but you can have several RPMs with the same name but different version.
On the other hand, working with the OS is much easier when the packages differ in their name, so this "workaround" is much more pleasant to use, even though it's not a "pure" solution.