Hacker News Reader

Comment by voltagex_ | Hacker News Reader

Has anyone got a checklist for package managers? There are so many it'd be nice to be able to predict what problems they're going to face in the future (and hopefully prompt better design in the process)

View on HN

Rules:

1. Use the system package manager.

2. See rule #1.

In all seriousness, there needs to be a good way to [spray your stuff](http://dilbert.com/strip/2013-06-09) all over a system with the rest of the system knowing about it. FPM does a decent job of this.

There is some research out there on common things that this kind of code gets wrong: https://www.cs.arizona.edu/stork/packagemanagersecurity/

This is especially true with native code, between apt/yum/zypper/pacman on Linux and pacman as part of msys2 on Windows you really should be using your system package manager (if you need to, build your own packages, it really isn't that much harder than any of these language-specific ones, I swear).

There was a tool I used years ago that would build rpm's from makefile-based build files. Basically, you would do './configure && make && [app name] make install' and it would then run a dummy 'make install', monitor syscalls that did copying etc. and make an rpm from that. That was easy enough to make packages from tarballs. Anything more complicated (like, having to actually know the package file formats) - forget it.

Anyone know the name of this tool? It was unmaintained for a long time already when I stopped using Linux, and that was 10 years ago.

I do remember using that tool to publish the very first aMule packages for Debian. I think it was called watchinstall.

It's called `checkinstall`: https://wiki.debian.org/CheckInstall

Yeah that's the one, seems it's maintained after all. It was/is awesome.

I used something similar named epkg, https://en.wikipedia.org/wiki/Encap_Package_Management_Syste....

It was probably checkinstall, today I'd really recommend learning how to make an .rpm or .deb the proper way, it's really not that hard. My experience with debian packaging is fairly limited, but the typical rpmspec for a autotools based build is as simple as (minus package metadata because that's kinda noisy):

  %prep
  %setup -q


  %build
  %configure
  make %{?_smp_mflags}


  %install
  make install DESTDIR=%{buildroot}

  %files
  /usr/*

Boom, there you go, build it with rpmbuild or mock and you're done. Debian control files are just Makefiles with a bunch of magic targets that get called, and debhelper 99% of the time will automatically figure out what you are trying to package (autotools, python, ruby, java, node, etc) and build/install/package it correctly for you so that most debian/control files are one or two lines.

You only even need to go the debhelper route if you're intending your package to go upstream into Debian or Ubuntu itself. You can go a looooong way with just `DEBIAN/control` and `dpkg-deb --build`.

> 1. Use the system package manager.

When system package managers get properly documented maybe ... when instead of having 1 package manager per system , we have a single package manager for every system ... otherwise no, use the package manager that comes with the language if there is one.

Which package manager are you running into trouble with?

> When system package managers get properly documented maybe

Invalid complaint. The major package managers are extremely well documented. Even some of the minor ones, like archlinux.

I don't think I'd call pacman minor anymore, it's pretty well known and is the package manager for msys2. slackware, on the other hand...

The day I am able to produce a package that will work in all OSes that I care, maybe.

Until then, I surely will not waste my time generating one package per OS/distribution flavor.

So you are developing a package in language and you package it with insert language specific package manager you are now done. If you package it for insert system package manager to achieve a similar level of coverage you must not only figure out how to create packages for 17 different systems you have to either figure out the standards and procedures required to get your package included on each platform or figure out how to host your own repo and instruct users on how to set it up this may include variations in setup and instructions that differ between multiple versions of the os that may be in use concurrently.

This could be 1000 x as much work for not much gain.

Right, much more work for little gain for the developer.

Then you have a bunch of developers all taking the easy way. And now all the system administrators running systems with those projects can't trust their packaging tools anymore, and have to do a lot more work in their daily workflow.

Oh, and as a bonus, none of these systems have a secure update mechanism, so you've bypassed the inherent code whitelisting that comes with a system package manager with a centralised repository and a requirement for strong package signatures. At best, they do like PyPi and optionally allow a developer to specify a GPG key. So you have to do "pip install <bla>" and pray to the computer god that the developer of <bla> and all the developers of all its dependencies cared about code integrity and set up GPG keys. From experience, the chances of that are approximately 0%, because it's not a requirement.

> So you are developing a package in language and you package it with insert language specific package manager you are now done.

No you are not. At this point, you have catered for developers in the language. You have not yet catered for final users.

Really, the Linux landscape is not that scattered. Package for Debian and Redhat and you have done the Pareto effort.

What about OS X? Windows? iOS? Android?

Ideally, your language package manager should know how to build (candidate) packages for common system package managers (eg '''setup.py bdist_rpm''' or '''setup.py bdist_debian'''). That way, the technical end user can install the package in a way their package manager knows about. If your package becomes widely used, it also simplifies (but does not remove) the job of the distribution's packager to build an official package.

This doesn't work so well for micro-dependencies, though. The TeTeX distrib. in Fedora is bad enough, I'd hate to see the same done with node packages.

This isn't really usable if you develop with the latest packages for dependencies. While I could generate an rpm for my python module, Fedora and CentOS/RHEL are going to have incompatible dependencies. I could package up my dependencies too, but then I'll break system software (a lot of fedora's tooling is written in Python).

A much better solution would involve making languages like Python accept the Python version as a dependency while making building the PYTHONPATH more flexible with common tooling. Making PYTHONPATH building better would enable installation with namespace+version, such that we could have:

    /usr/lib/pymodules/requests/1.0.0/...
    /usr/lib/pymodules/requests/1.2.1/...
    /usr/lib/pymodules/python/2.7.10/...
    /usr/lib/pymodules/python/3.5.0/...

Python in particular is still a bit challenging in that the compiler assumes it can spit out byte-compiled code all over the place (.pyc), but if someone was going through the tooling effort, they'd presumably find a way (or get something merged if it doesn't exist), to add a shadow directory of .pyc files that could be keyed by version and cached somewhere user-writeable (~/.pymodules for example).

The purpose of PEP 3147 is to fix some of those problems: https://www.python.org/dev/peps/pep-3147/

Addendum: I'd might as well mention that PEP 3147 is implemented from Python 3.2 onwards, so for Python 3, some of the problem is solved.

You still have the issue of being able to install different version of the same library.

That only really works in a world where all of the users on the machine that can run Python also can write to shared library directories. That seems kinda awkward from a security perspective unless you're already using containerization (in which case you don't care about any of this).

This is certainly better than it was (.pyc files just didn't co-exist sanely across python versions), but it doesn't provide a method for running multiple versions of libraries between different apps.

> That only really works in a world where all of the users on the machine that can run Python also can write to shared library directories.

Far from it. It's up to the system package manager to generate the .pyc files for the globally installed version of the libraries. __pycache__ directories allow this to be done cleanly for any installed interpreter that supports them. It has all the information it needs, and users need not get involved.

Unless the user is running a copy of Python that wasn't installed by the package manager, there should be no need for them to cause the .pyc files to be generated. Besides, .pyc files are an optimisation so that parsing and bytecode compilation doesn't have to repeatedly happen, and aren't even strictly required: the result is slower startup, but it couldn't prevent the code from being run.

> it doesn't provide a method for running multiple versions of libraries between different apps

Indeed, and I pointed that out in my comment in the addendum. Virtual environments are still pretty much the state of the art in that regard, unfortunately. It's possible that some PYTHONPATH magic might help with that to some degree, but that would be messy.

Not that it is ready for primetime yet, but guix solves this problem well by allowing each user/application to have its own dependencies of different versions while allowing them to still share if they want the same version.

May sound weird, but take a look:

https://www.gnu.org/software/guix/

The main distribution will only ever have free/open source software in it but it is trivial to make your own packages/package repositories and share them since it is just a guile code file.

Correct me if I'm wrong, but isn't multi-versioning one of the issues that eggs solve?

Or use CMake! CMake does this!

https://cmake.org/Wiki/CMake:Packaging_With_CPack

Do any system package managers support per-user installations, or are they designed mainly to install to system directories? Pip for example has the option to install packages into the user's home directory, which allows for easy cleanup. If developers want to experiment with a library for a project, they might not want everything chucked into /usr/ and made part of the system.

This is actually why I immediately discard use of the system package manager. One library version set per OS installation means that I can't run two things on the same machine unless I get really lucky with dependency versioning. Now the only way to get things done is to spin up one virtual machine per project, and completely setting up a new development environment per project is a huge pain in the ass that can only be optimized so far.

nix/guix is a sort of reasonable answer to this issue, I guess, but then you have to develop on nix/guix and hope that apt/rpm/pacman can handle your dependencies.

At least for guix, you can install it on top of any distribution and use it instead of the system package managers. I had some troubles with this when I did it, but nothing deal breaking and it isn't 1.0 yet so I imagine some of it will be cleaned up.

FreeBSD ports and pkg allow you to set the installation base, with $LOCALBASE and $PREFIX (I guess other BSDs do too). Also AFAIK Guix. And BSD systems use /etc and /bin and /usr/bin for OS stuff, and /usr/local for packages. Even the config files go to /usr/local/bin/etc (NetBSD same, but in /usr/pkg).

GNU Guix[0] supports unprivileged package management, among other features [1]. Each user may maintain one or more "profiles" containing whichever packages they'd like. On top of that, they have access to a universal virtualenv-like tool called "guix environment"[2] that can create arbitrary software environments for development or even just to do one-off experiments. It can even create containers for maximal development environment purity.

[0] https://gnu.org/s/guix

[1] http://www.gnu.org/software/guix/manual/html_node/Features.h...

[2] http://www.gnu.org/software/guix/manual/html_node/Invoking-g...

Generally the purpose of a system package manager is to install packages for the system. I think the GP's assumption was that the context is "how do I set up a system whose primary purpose is deployment of X", not "how do I write X on a system whose primary purpose is development".

Then again, I've had good mileage out of making .debs that plonk dev work and deps into /opt locally. YMMV.

All software (not just languages) evolves until it contains a half-arsed rewrite of apt-get.

You should be so lucky as to have a program package manner be at all compatible with apt or yum.

https://medium.com/@sdboyer/so-you-want-to-write-a-package-m... would be a good start

[jk] a package manager of package managers is coming to rule them all )

Will the package manager manage itself?

bootstrapping package manager, yes )

It will manage all the package managers that do not manage themselves.

package calculus

it should be called russell's package manager.

Don't most package managers do that themselves anyway? Like npm and pip seem to. Or are they doing something different behind the scenes.

Zypper, apt, dpkg, pacman, etc. all manage themselves.

Given how many non-obvious but easily-solvable problems there are with package management... this might actually not be a terrible idea. The biggest issue would be making it sufficiently flexible to integrate with anything that wants packages, including things like video game modding platforms and programming languages with in-language namespacing/module/package features.

Theoretically, apt-get/rpm can be a package manager of package manager. (e.g use them to install npm/qpm)