Hacker News Reader

Comment by jordigh | Hacker News Reader

Why do we keep running into this problem over and over again? Why do we need one package manager per programming language? Why is software distribution so damn hard?

View on HN

Because general purpose package managers suck at dealing with the concerns of individual programming languages. E.g. installation paths, virtual environments, different compilation methods, etc.

There is also no single cross-platform package manager. Mac has macports and homebrew, windows barely got one, linux has yum, apt and others. As a language designer, you don't want any part in this, you want your code to work. So you make your own.

edit: To clarify, I'm suggesting that we're stuck in a local maximum - now that all these systems already exist and have widespread adoption and tool dependency in their own environments, there isn't much of an impetus to use Nix or something similar even though in the grander scheme of things it'd be way nicer.

I think the solution is to separate the problem into two parts

- The "puts things on disk" part (posix "install" or a low level tool like dpkg or rpm is most like this)

- The "determine what needs to be installed" part, (aptitude, yum)

I'm all for a per-language or system version of the latter, but once that's done, you should generate packages installed by the former.

Heck, wrap all the commands (cp/mv/install/chmod/chown/etc.) that write stuff to permanent places on disk to actually do "add to a package", give it a basic name/version number, and have the low level tool handle adding/removing it from the system (or multiple systems, or deploy it, etc.). All the dependencies, compatibility, etc. are handled by the higher level system.

This gets you the best of both worlds - system level packages, and the ability to install whatever you need. FPM (https://github.com/jordansissel/fpm) is a pretty good example of this philosophy.

But, instead, we get every punk ass CPAN descendent spraying it's crap all over the filesystem, needing the whole build environment installed, touching the internet in weird ways that don't guaranteed repeatable behavior, etc. sigh

Haskell folks seem to be gravitating towards the Nix package manager, which is general-purpose. We need to stop the proliferation of a package manager per programming language and make systems package managers capable of handling the use-cases of pip, bundler, npm, cabal, etc. Nix and Guix are the only 2 general-purpose package managers that I know of that can do it.

I don't think it's gravitating "toward" Nix. Many members of the community are trying to use Nix personally and maybe Cabal proposals are inspired by Nix. I don't think the idea today would be that the entire community should move to Nix, however, not by a long shot.

> Nix and Guix are the only 2 general-purpose package managers that I know of that can do it.

For anyone constrained to GNU/Linux, right?

> For anyone constrained to GNU/Linux, right?

I don't feel constrained by a free software operating system. Besides that, a bunch of people use Nix on OS X. On Windows I suppose you're stuck with the inferior package managers.

"I don't feel constrained by a free software operating system."

I appreciate the sentiment, but there's a substantial difference between "constrained to" and "constrained by".

There are many other OS out there.

Why do people only think of the triad?

Nix works on OSX (although the packages themselves may not)

Wouldn't say that the community is gravitating toward Nix, it's kind of a false split since Cabal and Nix can only be used together and Nix can't replace cabal. Though Nix can be used to manage a wider package environment than cabal-install can and then cabal is left to the role of a Haskell build system which it excels at.

Thanks for the explanation. I'm not a Haskell programmer so I was basing my observation off of some Haskell hackers blog entries.

In my local group of haskellers, nixos is taking off as a system level package manager that does virtual envs, and multiple package versions pretty well. I haven't had a chance to play with it, but perhaps a sufficiently smart package manager can do both?

Because general purpose package managers suck at dealing with the concerns of individual programming languages. E.g. installation paths, virtual environments, different compilation methods, etc.

And yet all these nifty package managers can't even seem to get things as simple and long solved as permissions straight. Even autoconf sets permissions better than, say, pip, where I have to remember to check my umask before I "sudo pip install something".

To add to that, the concerns that operating system package managers have are very different from the ones that programming language package managers have.

With an operating systems package manager, the focus is on shipping working, uber stable code, which is unlikely to break someones system. This (imho) is due to two different factors: 1) Operating systems have to work, no two ways about it, if your OS is broken, everything else is broken. 2) Users of operating systems are not necessarily experts. If their package manager ships experimental code which breaks their particular system, they don't know how to report the bug to the central OS maintainers. I think this is handled somewhat by having different repositorys with different levels of stability, a la arch linux's [core/extra/community/testing/AUR], however at the end of the day, the main repository must avoid having breaking code.

With a programming language package manager, the focus is on having up to date features, and catering towards power users who, if things go wrong, can generally fix them, or at least know how to contact, and how to phrase their requests for assistance. This means that it's generally more acceptable to expect users of a programming language repository to occasionally be served broken code, as defects will be reported more rapidly, and more clearly.

I find it interesting comparing the two package managers that I use most often in my day to day computing experience: Cabal, and pacman (Arch linux). Pacman offers a single version of each library per repository, and that version is as stable and tested as the repository requires. This is in keeping with the spirit of an operating system package manager, as it allows power users to install unstable packages from more experimental repositories, but tries to serve as stable code as possible to general users. Contrast that which cabal, which basically offers no stability guarantees, but allows for much more fine grained control over library versions, sandboxing etc.

TL;DR

In my opinion, OS and Language package managers have goals which are at odds with each other: OS package managers want to maintain stability, Language package managers want to allow for bleeding edge code. The concessions that cabal makes towards stability (versions, sandboxes etc) often cause other problems as well.

The problem Cabal is trying to solve here is supposedly more difficult than that solved by Gem, Pip, and other package managers. GHC does some kind of cross-library optimization that causes dependency problems in routine situations.

I agree we shouldn't have one package manager per language, but I think so far no one package manager has proved sufficiently general and robust to serve all use cases. For example, I think something like Nix would be wonderful for all language communities, but it doesn't run on windows. That immediately takes it out of the running. And as far as I'm aware, all other package mangers would have the same "Hell" problems of cabal because of the GHC optimization mentioned above.

It is more difficult. By design Haskell shifts an enormous number of failure modes to compile-time and as such cabal gets an unnecessary amount of blame for library bugs. If you ``pip install`` a package that transitively dependencies on another library where the author has accidently introduced a backwards incompatibility in the API that occurs in 5% of use-cases, pip won't detect the change and will happily install the package and 95% of people will assume no problem exists until they hit a runtime failure. The job that pip has to do is trivial actually. In Haskell if there is an incompatibility that breaks the interface, and in the presence of "wild west" packages there will be, then it will manifest as a compile time build error instead of proceeding.

Not only is the package management problem more difficult, it's not the problem that Cabal (the Common Architecture for Building Applications and Libraries) was intended to solve! cabal-install is not a package manager; it can't uninstall packages or even record which packages it has installed.

Let's not forget they are all garbage from somebody's perspective.

I know Python. Would I prefer to write my package manifests in some bastardized Ruby DSL or Python? Of course I'd prefer Python and the Ruby guys would prefer their bastardized DSL. I don't blame them.

How do we bridge that gap? Python => Ruby => Python is bad enough. What about M4 or AWK or Makefiles or Haskell or god help us all some shitty custom format ala Puppet?

You might as well demand everybody on Earth speak English.

> You might as well demand everybody on Earth speak English.

That's working out quite well so far.

>> You might as well demand everybody on Earth speak English.

> That's working out quite well so far.

Not really. English is a minority language, spoken as a primary language by only about 5.5% of the world's population.

http://en.wikipedia.org/wiki/List_of_languages_by_number_of_...

One more juicy anecdote: there are more Chinese people learning English right now, than there are English speakers in the U.S..

Your postscript seems to confirm GP's point. b^)

I don't know that that's a sign of it working well...

Not if the students' plan is only to remain competitive with native English speakers. Knowing how to speak English as a second language, for pragmatic reasons, isn't the same as being an English-speaker in the sense of the parent post.

Why does the reason matter? It's the ability to speak it that matters.

You don't need to be an expert on the language used in your package management. 'second language' is a rather apt analogy.

> Why does the reason matter? It's the ability to speak it that matters.

Yes, I basically agree. I think there's a difference between a native speaker and someone who can order food in a French restaurant -- I mean, without getting snails when what he really wanted was escargot. But I agree, it's not a very important distinction.

Why are you considering native speakers to be the relevant statistic here, rather than total speakers?

Secondary language is good enough. As long as I am likely to be able to communicate with a random person on Earth, I'd say the goal is achieved.

I think a lot of it is that people are scared of automake, so they reinvent their own build systems, with varying degrees of badness.

Then they claim that theirs is "easy" but they've only done the first half of "simple things should be easy, hard things should be possible".

A properly autotooled package offers an amazing level of control about where to put things.

Some package managers are part of build systems too, e.g. the Maven ecosystem. Now in what language should you write extensions or plugins? There will be djihad.

Reply on news.ycombinator.com