Hacker News Reader

Comment by dom0 | Hacker News Reader

pip & similar package managers are a trainwreck everywhere but on developer machines.

As someone who uses Python on a Windows box, I can count a few times where I ended up having to find binaries that were pre-compiled, to install a package from Pypi. While that isn't the case for the majority of the packages I've used, I certainly wouldn't say it's something an end-user should/would do.

Same goes for npm, too.

Huh. I'm not originally from the python community, but I had no trouble figuring out how to get reproducible installs out of pip using the requirements.txt idiom.

That great! The PyPA works hard to make pip+pypi behave intuitively. That said, a lot of more applications need more machinery, yielding utilities like pip-tools[0] and conda.

[0]: https://github.com/jazzband/pip-tools

The issue with using pip and package managers in a similar vein is that they're not distributing complete packages, which usually means that stuff is compiled when installing some application. End users don't have compilers, and ideally you'd want to avoid them on servers as well. Often this also needs some or many OS-level depencies, like libraries and headers.

There are of course many other issues with pip and setup.py, far too many for a comment here.

But to give just one example; distutils/setuptools try to play god ^W build system with things like Extension. This works, badly. For example, most compilers that aren't GCC or MSVC compatible are not supported at all. Finding libraries? Not supported. Having proper dependency graph detection of files? Nope. If some "invisible" dependency of an output was modified, won't recompile. Ups. It pretty much behaves like a Makefile written by a naive person, just without the "-jN" flag.

Now I know that the PyPA guys do a ton of work to make all this stuff less brittle, and indeed, Python packaging has improved a lot over the years. But overall it's still brittle, complex, difficult to debug and reproduce, and generally falls apart randomly with a frequency that does not inspire trust.

There's not really a lot you can do about this though, is there?

It's not that hard to test that things work, or to see the failures. Comments I've seen like 'there should be a server that installs every package on its own and makes sure its tests pass' have a bit of an issue: we already know the issues exist. Having a big list of them in one place doesn't really give us a lot more information.

What's the solution? If you need to compile native dependencies, how do you take PyPI and Python packaging in general and make those native dependencies able to be portably compiled, reliably, on any platform?

Things that are straight up unworkable:

* Bundling native dependencies as binaries * Specifying a single version of your native dependencies (the 'shrinkwrap' approach) and every library having its own

> Building native dependencies as binaries

Yep. http://conda.readthedocs.org

What? I said pip works great in my environment, not that it works great in yours. By all means, if you're concerned about those cases, use native packages or whatever you like.

I don't intend to contradict you, I'm just providing my angle on where the original comment came from.

Python does now have a distributable binary format: wheels (.whl files). A wheel contains everything already built for the target operating system/architecture, and installation consists solely of moving files into their destinations.

Also, I don't have the raw numbers to know for certain, but I suspect packages which have compiled extensions are more rare than pure-Python packages.

Reply on news.ycombinator.com