Hacker News Reader

Comment by sergiosgc | Hacker News Reader

Package management is probably, today, a very well defined problem. Isn't it time to solve this once and for all? Or are we forever doomed to have multiple package managers for the dot product of [operating system]X[development platform]?

It itches me that php has pear and composer, node has npm, ruby has gems and python has pip. It itches that deb and rpm solve the exact same problem, not to mention all the other smaller package formats/managers like pacman, yaourt or portage. It looks as pretty obvious that it is the same problem, albeit attacked via some different angles (source build versus binary distribution notably).

It even looks like it could be cross-platform, solving in one simple swoop at least unix systems (namely linux along with the BSDs) and, in a dream world, non-unix systems (windows, notably).

Daydreaming, yes. The status quo is too well established. But it justifies my jaded look regarding Guix: Late comer to a party that is already stale and over.

View on HN

Nix is attempting to be the solution you're looking for. Look at the nixpkgs repository: https://github.com/NixOS/nixpkgs

You will see packages for tons of stuff, including:

    * Python packages from PyPi: https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/python-packages.nix
    * Haskell packages from Hackage: https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/haskell-packages.nix
    * Perl packages from CPAN: https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/perl-packages.nix
    * NodeJS packages: https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/node-packages.nix
    * Lua Packages https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/lua-packages.nix

All these packages in one place, all installed through the same system and automatically built and tested on the Hydra build farm. Nix users have developed tools which automate (as much as possible) the creation of these packages.

I really believe NixOS is the future of the Linux distro, and it's tragic that it's still relatively obscure. It revolutionizes the OS in a myriad of ways, getting everything into the same package repository is just the beginning. For instance, it's fully configuration managed out of the box, using the Nix package manager as its configuration mangement system. Puppet, ansible, saltstack, chef, etc. are all made obsolete. Sure, it still has some rough edges, but it's a great community to contribute to and is improving at a impressive clip. If you're interested in the other ways NixOS is better than whatever you're currently using, I refer you to the website: https://nixos.org/

Just one question: how much it takes to run Bumblebee, zsh and Steam on that?

How easy is it to build your own packages from, say, the tip of the master branch of github repository? How automated is building the dependencies?

Building packages on Debian like systems is a huge pain. Arch is much better but I need Debian/Ubuntu-like reliability.

Very easy.

    git clone git://github.com/NixOS/nixpkgs.git
    cd nixpkgs
    nix-build -A packagename

Building the dependencies is completely automated. For more information, see the nixpkgs contributor's guide: http://nixos.org/nixpkgs/manual/

Note that the above command will simply download cached binaries where they are available, but if you make any changes to the build scripts nix will notice rebuild locally instead. This gives you the benefits of both source based and binary based package managers.

Is there no Ubuntu 14.10 nix package?

Guix is based on / inspired by Nix, but using Scheme instead of a custom DSL.

> curl https://nixos.org/nix/install | sh

Installation instructions for nix. Although I'm not sure that arbitrarily running shell scripts from the internet is a good thing. When I install an OS X app from the internet, at least it's signed.

That script does a lot of work that boils down to "go to https://nixos.org/releases/nix/nix-1.7 and do the obvious"

Of course if thats not signed you still have a problem. Eternal chicken and egg issues. How do you install gpg to verify a gpg signature while verifying the gpg signature of gpg

> Package management is probably, today, a very well defined problem.

I don't think it is. I think "package management" is a name for a general class of solutions that addresses a range of problems (varying, at a minimum, along a continuum between "I want a way to pull a set of implementations of particular capabilities on any of a variety of different systems for which the appropriate implementations of those capabilities may be different" to "I want be able to reproduce from a list the same set of implementations across different machines".)

There's also a range of priorities about certainty of results versus freedom to mix and match and ease of contributing to the globally accessible repository.

I very much don't think the problem addressed by package management is "well-defined", and I'm not even sure it is coherent in the sense that were the problems different people look to package management to solve all collected, there would be a single workable solution that would address all of them optimally.

The imperfect analogy I've used is the comparison between paints and pencils. They perform similar general functions (placing something on a media) and even use a similar set of tools (a long stick held in the hand). But the end result and even the goal could be totally different (I may want to hang a painting on a wall in a frame and I may want my pencil drawings to be reproduced a book).

It's not perfect, but it helps non-users (i.e. managers) understand that one tool does not fit all.

Guix is not late in that there aren't many purely functional package managers out there. It's the second after Nix if I'm not mistaken, and it doesn't reinvent the purely functional wheel either; it builds on top of existing core parts of Nix. Why not use Nix? Most importantly because it's not GNU; NIH is sadly a real problem for GNU. But Nix also uses C++/Perl and a stand-alone package recipe language; Guix uses Scheme as much as possible. ISTR Guix also has some features that Nix lacks, such as unprivileged package management. Don't quote me on it though; mayhaps it's trivial to add to Nix or has already been done.

Cooperating with existing infrastructure is, I imagine, very difficult for a purely functional package manager, given the whole "purely functional" aspect. :-)

It's also extremely unlikely that Debian will just give up apt, and RedHad will just give up rpm.

And every programming language platform seems to have its own package manager precisely because of that reason: there is no one package manager on all systems, but a proglang platform wants to offer package management on all systems, so each bake their own implemented in and special-made for the language.

It's a sad state, but that's the world of software for you.

http://xkcd.com/927/

Nix does have unprivileged package management via a CLI tool, though the absolutely fantastic describe-the-whole-system-in-one-file wonder that's configuration.nix[1] takes elevated privileges.

[1] E.g: https://github.com/Fuuzetsu/nix-project-defaults/blob/master...

>Guix is not late in that there aren't many purely functional package managers out there.

Why does almost anyone not involved in writing it care what language the package manager is written in?

>Cooperating with existing infrastructure is, I imagine, very difficult for a purely functional package manager, given the whole "purely functional" aspect. :-)

I don't follow. Why is that difficult?

>http://xkcd.com/927/

You can have different implementations of the same standard. To make your own standard is a step beyond.

Being functional has nothing to do with the language it is written in.

http://www.gnu.org/software/guix/manual/html_node/Introducti...

"The term functional refers to a specific package management discipline. In Guix, the package build and installation process is seen as a function, in the mathematical sense. That function takes inputs, such as build scripts, a compiler, and libraries, and returns an installed package. As a pure function, its result depends solely on its inputs—for instance, it cannot refer to software or scripts that were not explicitly passed as inputs. A build function always produces the same result when passed a given set of inputs. It cannot alter the system’s environment in any way; for instance, it cannot create, modify, or delete files outside of its build and installation directories. This is achieved by running build processes in isolated environments (or containers), where only their explicit inputs are visible."

Of course, this approach has some disadvantages:

http://www.gnu.org/software/guix/manual/html_node/Security-U...

"when a package is changed, every package that depends on it must be rebuilt. This can significantly slow down the deployment of fixes in core packages such as libc or Bash, since basically the whole distribution would need to be rebuilt."

Oh, sorry, I got confused by the mention of Scheme.

Though that means the XKCD comic doesn't apply. This is a standard with a very specific goal, and it very much does not want to incorporate a lot of the earlier standards.

The comic was applied to the situation somewhat liberally. :-)

I guess you could say it's about de-facto standards here, rather than standards. Deb, RPM, pacman, Nix, etc., as well as all the language-specific PMs all solve the broad problem of package management, so one of them could become the de-facto standard and embrace the others, but it ain't gonna happen.

Thanks for the clarification. Further clarification:

I need to refresh my knowledge before I'm sure, but I think that statement regarding dependencies only goes for build-time dependencies. An update to a run-time dependency wouldn't require a rebuild of the dependent.

And the build-time dependencies' up-to-date status is not very important most of the time. "Does it matter whether I used Bash 4.0 or 4.2 to run that ./configure script?" Nope. I'm not sure to what extent this carries over to practice so far; the recent security patches to Bash did cause a total rebuild in Guix. I hope that will change by the time 1.0 is released. Theoretically, you could use the same build-time-dependency packages for years (e.g. a "bash-for-building" package), yet quickly deploy bleeding edge versions to users (proper "bash" package).

> "when a package is changed, every package that depends on it must be rebuilt. This can significantly slow down the deployment of fixes in core packages such as libc or Bash, since basically the whole distribution would need to be rebuilt."

I'm just wondering, wouldn't it be possible to have an additional checksum for just the public API/ABI/headers of the package? Would this be enough to at least eliminate cascading rebuild for upgrades to shared libraries?

"Purely functional" here doesn't refer to the language the package manager is written in - it refers to the package management itself.

In Nix (and therefore I assume Guix), this means that each package is built in an environment that only contains the package's explicitly-declared dependencies (to prevent undeclared dependencies) and is not modified after installation. Therefore, the result of the installation is (or should be, anyway) a deterministic function of its dependencies and the install process. Nix thinks of a package as a function whose input is folders (containing its dependencies) and whose output is another folder.

This also means that Nix package installs should always be reproducible across any number of machines, even if the machines contain different sets of packages. It takes some of the benefit of using a single master image for the OS (i.e. CoreOS), but lets you mix and match parts of the system.

I remember looking at Nix briefly a while ago. As far as I can tell, the difficulty of this approach is building system-wide configuration files that need to know about which packages are installed (sadly I can't think of an example right now), because a thing like that can't be installed once, as its own package, and then left alone. However, simple executables and libraries are very easy to model in Nix.

Late clarification because I wasn't careful and forgot that "NIH" has very negative connotations:

When I say "NIH is sadly a real problem for GNU," what I mean is that NIH can be an actual valid reason for GNU to fork or reinvent something, not that they're guilty for doing so. As in, it's unfortunate that they'd have to do it, but sometimes they have to. (Often it's not "reinvent" at all though, instead merely maintain an upstream compatible fork such as IceCat or Linux-libre.)

The reason is that if a non-GNU project is heavily relied on by GNU, the project might one day decide that they don't exactly agree with GNU anymore, so it's best to have a fork under GNU which can be fully trusted. Sometimes I find this non-trusting stance sad, but it seems to be necessary; just see Mozilla adding DRM to Firefox. :-(

I would argue that it's far from a well defined problem. deb/rpm have very fundamental differences from gem/pip/npm etc. CoreOS takes a very different approach to packaging, and so does Nix/guix.

If it truly was a well defined problem, everybody would be using RPM. All the package managers you mentioned were created because somebody perceived a problem or limitation with previous formats.

> I would argue that it's far from a well defined problem. deb/rpm have very fundamental differences from gem/pip/npm etc.

I know the internals of pear, pip, rpm and deb. I don't recognize fundamental differences between them, although this evaluation depends on the definition of fundamental.

At the very core, all of them catalog files on the filesystem, script configuration of files once written and trigger reactions from other packages to the appearance of a new package.

The key difference is one of domain. Development platform specific package managers do their operations from inside the development language, with the niceties that come about from that (better APIs to change the development environment). This is not fundamental.

As for being well defined, I state the problem as: If you lock in a room, for a couple of hours, the lead developers of deb and pip, would they be able to come up with a unified package manager? It's, for me, clear that they would. Rinse and repeat, and you get a full unified spec.

"It's, for me, clear that they would."

I highly doubt it. The deb developer is a sysadmin. The pip developer is a programmer. That's a big gap.

For example, you can install many different versions of a gem simultaneously, which is very useful to a programmer. A deb can only install a single version at a time. That's best from a sysadmin perspective.

You can have multiple versions of the same deb package. Look for installed Linux kernels on your system and you'll likely find an example right there.

This is actually a good example of the good definition of the problem space. All corner cases have been dug up. It's not a priori obvious that an operating system needs multiple versions of a package, but in some cases it does. And the package manager supports it.

You can have multiple versions of the same deb package. Look for installed Linux kernels on your system and you'll likely find an example right there.

Not so much - they have to have different names. I couldn't have a package named, e.g., linux-kernel and install version 3.4, 3.5, etc, instead I have to install linux-kernel-$version, each of which is a different package.

You're getting caught up in implementation details. It fulfills the functional requirements in debian, shows that the problem is common to all package managers, it reveals that this corner case has been touched by deb and in the end reinforces my original points that:

a) The requisites for most package managers out there are not that different (i.e. it's the same problem).

b) We have the problem so beaten down that we should be able to properly define the solution boundary.

Is the deb solution suboptimal? Probably. It's not relevant to the main point being argued.

>You're getting caught up in implementation details.

But if a package manager is anything, it's an implementation. It's not a theoretical concept, it's a tool that a user has to smooth the process of their work. Both could just install and configure packages by hand but it's the implementation that makes a package manager useful (for one being able to run multiple versions and for another being explicitly discouraged from multiple versions). Dismissing implementation details in a tool that is designed for implementation is missing the point.

In the obviously far superior ;-) RPM world it's possible to have multiple "linux-kernel" packages with different versions installed at the same time.

This is a pretty common point of view, but I wonder, how would this be enforced in a free-and-open software development ecosystem? Who would go around stopping people from launching their own take on a solution to the same problem apart from the "blessed" solution? In a world where everyone is free to choose which problem they'd like to solve and how to go about it, and it is impossible for everyone to agree about even the most trivial of options, you cannot ever have only one text editor, package manager, init system, etc.

Seeing how well most developers welcome change in their favorite OSes, editors, etc, I imagine a world where every developer bends their efforts towards the one true chosen solution, and are not allowed or able to choose to "waste" their efforts on another solution wouldn't yield as vibrant a software ecosystem as we have now.

> This is a pretty common point of view, but I wonder, how would this be enforced in a free-and-open software development ecosystem?

It wouldn't be enforced. The beauty of OSS is that there is no enforcing of solutions. Even when you touch something as core as the init system or the graphics system, the solution must prevail by technical merits. There is marketing, but thankfully this is a world of technical evaluation first, selling second and never one of enforcement.

Change is more difficult, but so are mandated blunders.

Nix (and by extension Guix) are meant to be this solution you're looking for. The problem, as with most unified solutions, is adoption.

Then that is the very first thing they must state. Instead, the opening line is: "GNU Guix is the functional package manager for the GNU system and a distribution thereof."

In addition, this problem can only be solved by luring established platforms, and this can only be achieved by providing a transition path. It should be able to coexist with other package managers (composer, or pip, or npm or, if shooting for the moon, rpm and deb).

Having multiple system package managers coexisting is pretty thorny. If very carefully delineated so they manage completely independent bits of software installed in their own prefixes, I could see it working (with some careful path management). But if you're actually installing/upgrading system software with it, having some mix of apt, rpm, and guix managing your packages sounds like a recipe for pain.

Much like how many of us use pip, gem, npm, composer, etc. on our Debian/Red Hat/etc. machines, you can also use Guix in the same way. For example, you are deploying a production web application and Guix had newer versions of some dependencies than your stable host system. You could use Guix to create a profile (a collection of installed packages, a bundle in Ruby terminology) that includes those dependencies for your web application to use.

It's possible to use pip and friends on a Guix system, too.

I personally do all of my Guix hacking on a Debian machine. They play very nicely together.

I'm not sure it really matters that rpm and deb solve the same problem unless you use different distros.

However npm, gem, pip, composer mostly solve the same problem, but a different one to rpm, in that we want per-project dependancies rather than per-system and want them to be commitable. Also many project combine languages so it would be good if these were combined.

It's well defined, but there's no good solution to this problem yet! See: http://www.well-typed.com/blog/2014/09/how-we-might-abolish-... It's true for every language out there, not just Haskell.

Don't forget the classic non package maintainer mistake of thinking policy = package file type/format.

Its pretty trivial to "force" a rpm to install on a Debian box. That doesn't mean that magically any of it will follow Debian policy, any of the paths will be correct, any of the dependencies will be correct. But technically files can be thrown onto the filesystem, even if they can't actually be used where they land. And vice versa, if you shove a deb onto a Redhat box.

My biggest "NIH" surprise is GNU has this and the emacs package system and at first glance they have nothing in common. The idea of installing a local copy of Rails or Leiningen off Emacs MELPA is kinda weird. I wonder if emacs twenty...something will use a user level version of this system. That would be interestingly consistent.

Guix is a big effort at making a purely functional package manager with a large recipe database and all kinds of bells and whistles you want in a full operating system.

Emacs's minimal package manager is to install Elisp libraries (and nothing else) on every platform where Emacs runs, including MS Windows, and you can be happy it does dependency tracking.

The two can't really be compared. Maybe in 10 years Guix will run on all platforms and have its components separated in such a way that it's sensible for Emacs to ship a version of Guix with it for Elisp package management, but don't hold your breath.

So I don't think NIH applies there.

I hope this solves the fad that every new programming language writes its own package manager.

Adopting a GPL'ed tool written in Guile to install packages for their own languages? I don't see it as likely.

GPL'ed means everybody can potentially use it. It's a stand alone tool the user has already installed, it does not have to be distributed, or liked against. And I don't see why these people wouldn't write a few lines a guile. Right now they usually write a few lines of shell.

I'm always confused to see this point brought up. Whenever there are lots of options, folks talk about the harms of fragmentation and wasted effort and focus. Whenever there aren't a lot of options, it seems like the same crowd demands disruption and competing choices.

Different people, probably. I'm all for consolidation whenever the problem space is very well defined, and divergence whenever the problem space is ill defined. This is a case of the former.

Package management may be defined but maybe not well named. More and more when I see package I see language stack, lexical scope and such...

Maybe the Systems vs Languages split is detrimental to the solution here.

>are we forever doomed to have multiple package managers for the dot product of [operating system]X[development platform]?

Did you, by any chance, mean Cartesian or direct product?

Mandatory XKCD comic: http://xkcd.com/927/

You beat me to it. :-)

Reply on news.ycombinator.com