Hacker News Reader

Hacker News

Comment by antocv | Hacker News Reader

Full thread

antocv

Can we get a package manager for package managers?

There is so many already, its hard to keep track what names come from what repositories of what package managers.

It used to be so simple, yourDistroPackageManager install whatever, now there is x2000 versions of packageManager.

View on HN

Reposting an old comment of mine on the subject::

The path to a standard package manager starts with a standardized protocol for package management.

A service protocol that is able to serve a repository of packages over http and ftp. A client protocol that can keep track of installed packages and can index, search and look for updates on installed packages.

Split package management into layers and only try to standardize bit by bit. People will never agree on deb vs rpm. People will never agree on using json vs python vs Makefile vs ruby vs shell vs whatever else - they'll always want their most familiar language for their package manager, which in domain-specific packaging means the domain-specific language.

So don't try to standardize those. Standardize the rest. Give us the protocol that can power all of this and increase interoperability. Separate the repository layer, the package format (deb, rpm), the packagefile format (setup.py, Makefile, PKGBUILD) and the package manager (interface: yum, apt-get, aptitude, pip, npm) from the rest of the protocol.

Make this potentially usable for things such as browser extension repositories, android package management, vim bundles and what not.

Someone please work on this. I'd do it but it just occured to me I have to clean my oven.

Well, since it's an old comment your oven should be clean by now :)

I'm working on something along these lines:

https://www.packagelab.com/

The idea is to have a common UI and REST API for building packages and managing repositories across packaging toolchains. Right now it supports rpm+yum and deb+apt. Packages are built in isolated Linux containers using Heroku's buildpack interface. (Would love feedback from anyone who's interested in beta testing.)

Also, have you seen the Omaha protocol?

https://code.google.com/p/omaha/wiki/ServerProtocol

It's used by a lot software, including Chrome and ChromeOS. Recently, CoreOS has built infrastructure with it and they offer a SaaS product for rolling updates. I think they're planning on using it to update Docker containers, too:

https://coreos.com/products/coreupdate/

I'll send you an email.

> Someone please work on this.

The only result will inevitably be that we'll have yet another package manager.

You completely misunderstand the point. "Another package manager" doesn't matter. People can and should use the package manager of their choice (why should I break pip?). The point is to have common ground between all package managers.

Could you talk about the advantage(s) you perceive of a "common ground between all package managers"? After using npm a bit I think scripting hooks in a package manager that can call other package managers enables many good things.

One is that you can easily specify dependencies on packages not written in the language your package is written in, in a way that's machine-readable. Imagine if I could do "npm install some-framework" and it'd automatically build and install the relevant versions of node.js and MongoDB as well.

This is what I meant by "scripting hooks". I have something like the following in package.json for various projects:

  "devDependencies": {
    "bower": ">=1.3"
  },
  "scripts": {
    "postinstall": "bower install"
  }

This way I never need a global install of bower, nor do I ever really need to use it directly. I don't see why something similar couldn't be set up for any other npm-able package manager or build tool. A bit of googling seems to indicate that npm can update node using e.g. the "n" module. That module or another like it can of course be a dependency in package.json.

This isn't machine-readable, for a start - I can't see what your package actually depends on without reading through your postinstall script. I also can't write a script to convert it automatically to a .deb or a NixOS package or whatever else, because I can't figure out what its dependencies are. I can't easily say "I want this package and all its dependencies on a DVD".

The dependency on bower is stated explicitly. The dependencies that bower manages itself are easily discoverable in bower.json. I don't see why a call to bower is any more of a problem than any other build step, e.g. make.

I get that having a standard format (let's say "dependencies", "devDependencies", and "peerDependencies" as properties of a json file) would make somebody's life easier, but it's not clear that is the same set of somebodies who develop and use all these package managers.

> I don't see why a call to bower is any more of a problem than any other build step, e.g. make.

Because dependency management is a separate problem from building. I need to be able to see what, exactly, every package depends on, and to be able to operate on that, to be able to deploy my system, and in order to know what exactly it is that I'm running.

Figuring out what my dependencies are at the same time as building my system leads to near-constant breakage, and I couldn't even figure out "am I using this package somewhere or not?" because there's no central package database when using npm/pypi/whatever else. How am I supposed to figure out whether my system is secure if I can't figure out what's installed on it to match against the CVE databases?

Today, I solve this with an extremely hackish set of scripts (npm2nix, pypi2nix, etc) and a lot of manual overrides to specify dependencies between bits of code written in different languages, and wrap that into one package database. Operating system distributors are forced to do the same.

> I get that having a standard format (let's say "dependencies", "devDependencies", and "peerDependencies" as properties of a json file) would make somebody's life easier, but it's not clear that is the same set of somebodies who develop and use all these package managers.

As a user of package managers, it would make my life easier if everybody would actually define all their dependencies in such a way that I can run an automated tool against a list of packages and discover every single package, so that I can wrap them into a DVD for offline installation, or figure out whether I'm vulnerable to something, or convert them for use with my own package manager, or whatever else I want to do.

Well, for one thing, new package managers could be implemented without having to worry about the infrastructure. The server software could be shared between all linux distributions and would be applicable for any sort of "bundle server"; think game addons, firefox/chrome extensions, skins for your favourite software and what not.

Package management needs to be solved at the systems level. Rather than having a new package manager for every language and every extensible program, the problems that make our systems package manager (apt, yum, etc) unsuitable for the task need to be addressed. The biggest problems to me are unprivileged package management (no sudo) and support for multiple "profiles" (think virtualenv, rvm, gemsets). Projects such as GNU Guix or Nix solve the problems quite nicely, IMO. We should develop them more instead of inventing more special-purpose package managers.

"Projects such as GNU Guix or Nix solve the problems quite nicely, IMO."

So long as you define the problem space as "package management for POSIX systems," and in the case of Guix you can scratch OS X and I think the BSDs off the list, too. If your package manager doesn't run on all of Windows, Linux and OS X at a bare minimum then your package manager is not solving the problems of Node.js, Python, PHP, Perl, Ruby or any other cross-platform runtime.

Sigh. Well fine, if you're stuck on Windows then language-specific package managers are much better than nothing. But on POSIX systems, we can do so much better than the current mess.

Package management at the systems level is a completely different problem from package management at the application level. An externally-facing server application is ideally a self-contained system of its own which should be installable on any modern kind of system without regard to whatever is or isn't on that system already, beyond the bare minimum. Trying to resolve library dependencies between two externally-facing applications is pointless; there should be no effect of one on the other.

In the days when space and bandwidth was expensive, a lot of compromises had to be made, but with many current language ecosystems, the cost of maintaining separate infrastructure is trivial, so we shouldn't be continuing to make our lives more difficult by deferring to the system package manager.

The sibling comment brings up Nix, but the only reason that looks like a good solution is that it happens to make all libraries application-specific libraries (if I understand correctly), which fuses the two problem domains. Nix excepted, however, I think it's a mistake to conflate them.

Ugh. I think there's a lot of sense in this comment, but it still seems horrendously wasteful to install multiple, redundant versions of libraries just to maintain separation between applications.

Hell, space and bandwidth are cheap: Let's just distribute all of our applications as 2 GB binary VM images.

Reply on news.ycombinator.com