Hacker News Reader

Comment by dice | Hacker News Reader

As a sysadmin pip and its ilk annoy me as well, although for a reason which is not mentioned in the article: It creates an entire package management system which is not the distribution's package manager. Ruby's gems tick me off for the same reason. As a sysadmin you need to decide to either manage Python and Ruby entirely outside of the OS's native package management or try and wrap every single Python egg and Ruby gem in an RPM/DPKG/whatever. Mixing the management between two packaging systems is just going to cause trouble.

There's a similar problem with Perl's CPAN, which is what I think all of these projects are aiming to emulate, but the nice thing about CPAN is that repositories like rpmforge or EPEL already have a large number of commonly used CPAN packages all wrapped up in RPMs already. It would be nice to see similar community efforts centered around Python and Ruby packages. Perhaps one day I will have enough free time to start one.

View on HN

The problem is the distributions packages are often horrendously out of date. I have stopped using them entirely even for things like Numpy and Scipy which I would really rather use them for (as they take a long time to compile and rely on several C libraries and a fortran compiler). It isn't my favorite thing to have my fabric task compile all the dependencies but there isn't really an easy way around it.[1]

In general I think pip has drastically improved my work flow and code structures. For instance: I no longer use submodules for my library code, I pip install the git repositories. Huge improvement, way easier to manage.

[1] Obviously, you can get around it if you do things like copy the virtualenv and fix the paths as in the OPs post. However, I wouldn't call that "easy".

It's true that the distro's packages can often be out of date: particularly with "Enterprise" releases like RHEL/CentOS. That doesn't mean that one shouldn't use the package manager, though. Building custom packages to backport updates and tracking the upstream for security and bug fixes isn't the funnest thing in the world, but it is often a necessary evil. It's certainly better than a "compile from source then dump everything into a tarball" approach which leaves you with no good way to track what versions of which software are installed on which nodes.

I think that a 'best practice' is to do something like create a virtualenv for your app and package the entire virtualenv in the distro's package manager, then version that.

That doesn't mean that one shouldn't use the package manager

Using the distro package manager for eggs, gems or jars is an exercise in futility.

Bundler, Maven, NPM et al are not flawless, but maturing rapidly, and a much better fit to the problem domain.

It isn't a necessary evil when it isn't necessary. If you have isolation (like virtualenv) then each app gets exactly what it needs. Life is too short to waste it trying to make every single bit of Python use the same versions of dependencies, pinning the version in the package manager, etc.

The universe does not revolve around lazy sysadmins

Ubuntu is pretty good about keeping it's packages up to date and specifically Numpy and Scipy.

I don't understand why they can't all play along. Why can't pip packages be generic enough to be transformed into rpm/deb packages? Why can't a repository of these packages be maintained with a comprehensive and up to date selection of packages for each OS?

Why must everyone reinvent the wheel, but do so in an incomplete way?

The answer, as far as I can tell, is that it makes it easier for Python (and Ruby and Perl) developers to share code and stay on the "latest and greatest" versions of their libraries. I mean, as a library author, what would you rather do? Just create a single egg, gem, CPAN package, or whatever? Or would you rather create three or four versions of the same package (one for Debian-based systems, one for Fedora-based systems, one for Gentoo, one for Arch, etc.)? I understand that it can be frustrating from a sysadmin perspective (pip, especially, isn't sysadmin friendly at all), but from the perspective of a library author, eggs and gems are much preferable to debs and rpms because I know that they'll work in a reasonably distribution independent fashion.

EDIT: I know that's not a satisfying answer. But, as far as I can tell, it is the primary reason that runtimes like Python and Ruby include their own package management rather than falling back on the package management that the OS provides.

I don't agree with that approach. It sounds nice for a sysadmin, but as a programmer I don't want to have to wait for you or the distributor to approve and package the libs I need to work with.

>I don't agree with that approach. It sounds nice for a sysadmin, but as a programmer I don't want to have to wait for you or the distributor to approve and package the libs I need to work with.

Which is why development, QA, staging and production are all separate environments. Do what you want in your dev environment: when the code is ready for testing let the ops team know what the dependencies are. We'll take care of wrapping everything in packages and updating chef/puppet/whatever to deploy it.

This doesn't consider the use case of needing to have multiple versions of the same package installed for testing. Sure, it would be nice to be able to have a second machine provisioned to do testing on.

And, as a extension of that, if you are using virtualenv+pip for testing, it makes sense for the deployment to use that too, as that means that the testing environment is closer to the deployment environment.

I have the same problem, fortunately there is fpm. I use it to create RPM binary packages from pip repos -- problem solved.

First I've heard of it, thanks for the pointer!

Reply on news.ycombinator.com