Hacker News Reader

The problem is that, as a software developer, I am not going to learn 3 PMs to distribute my free software (yum, apt, homebrew) nor am I going to navigate the politics of Fedora, Debian, Ubuntu, etc. for getting my free software included in their repositories and/or keeping it current. I have things to do, like write software.

Language PMs are "good enough" to solve the problems of the people actually writing the software, and in most cases that is really all that matters. I'm sorry they're not good enough for you, but it's not my fault that Rust, Python, and Go will let me upload to their package manager but Debian/Fedora/whatever you use will not.

Your fight is not with the language PMs, it is with your OS vendor who will neither give me upload rights nor send out someone else who does, who neither designs tools for me to use nor sends someone who already knows how to use them.

To be clear, I think there are good reasons why OS vendors won't change their ways, but software developers won't change our ways either, so neither language PMs nor OS PMs are going away any time soon.

No one is stopping you from creating your own yum or apt repository. For Fedora you could use COPR [1]:

> Copr is an easy-to-use automatic build system providing a package repository as its output.

> Start with making your own repository in these three steps: 1) choose an architecture and system you want to build for; 2) provide Copr with src.rpm packages available online; 3) let Copr do all the work and wait for your new repo.

[1]: https://copr.fedorainfracloud.org/

You missed his point. As a software level developer if he uses "his language package manager" that's where his responsibility stops. In your example, he'd make a package for yum or apt but what about brew what about $obscure_other_one? It's an abstraction layer.

The responsibility to package software at a system level isn't yours, it's the system vendor's. Let them hash out a standard packaging model while you ship your plain tarball or provide `configure && make && make install` instructions.

> Your fight is not with the language PMs, it is with your OS vendor who will neither give me upload rights nor send out someone else who does, who neither designs tools for me to use nor sends someone who already knows how to use them.

As far as Fedora is concerned becoming a package maintainer is a fairly straightforward process, and you are more than welcome to package up your own software and submit it to the software collection or solicit someone to assist you with it. The rpmdevtools package contains a super-useful utility called rpmdev-newspec that provides templates to create packages for most popular standards out there already (ruby gems, python setuptools/distutils scripts, CPAN modules, etc) - it's really easy to get started and you can always push your packages up to COPR without needing sponsorship from the existing packaging team.

This is missing the point. The problem is still that there is a million different system PMs, but (usually) only one for each language. The package maintainer have no incentive to care about Fedora, regardless of how easy it is. The real solution is not for Fedora to absorb all language specific packages but to integrate the different package managers so that they work better together.

> The real solution is not for Fedora to absorb all language specific packages but to integrate the different package managers so that they work better together.

Which is precisely why rpmdev-newspec has templates for this case. Simply running rpmdev-newspec -t ruby/python/perl/php-pear/ocaml/R mypackagename will generate a rpmspec file ready to go for any of these languages using their standard package managers (gem, distutils/setuptools, CPAN, pear, OPAM, packrat) - just add necessary metadata (Requires/BuildRequires, summary, description, version, and a changelog).

It's not magic, it still requires minimal effort (3 minutes worth of metadata + calling mock to test the build) - but it's hardly difficult. Sure, you could make some extra scripts to automatically populate the metadata from the language PM's descriptor of choice to limit the manual work to release bumps and editing the changelog - and I don't think anyone would be opposed to that, but that's pretty much all that would be left.

The biggest problem is outside of rolling-release distributions like Arch, Gentoo, OpenSUSE Tumbleweed and Fedora Rawhide bumps in major versions within a release are highly discouraged - and a lot of people used to using language PM's have a habit of changing things rapidly even post-1.0 and not supporting old branches - often leaving the distribution packager with needing to backport patches for security or bug fixes. I'm not going to argue if it's better or worse, but it's an issue that needs to be addressed.

The ideal solution would be for the system PM to delegate the responsibility for the language dependencies to the language (preferably compartmentalized). It would be pretty slick to have a standardized way for language PMs to be able to declare dependencies on system packages.

What you say is very sane why should learn 3 packet managers. The problem is that language packet managers are not good enough, they are enough to get software to install on the developers system. Often leaves the future and the users hanging there with something less optimal.

I guess my issue is that "good enough" is a very fluid concept, and I often discover some months after that it was my excuse for shitty code/integration test.

> The JVM world has good things going with Maven and Gradle.

Gradle internally uses Maven. Leiningen (the Clojure build tool) also uses Maven, and the nice thing here (while being rife with other problems) is that it pulls the correct language version in as a dependency (i.e. it pulls in Clojure in the required version).

These were my thoughts entirely while reading the article. If I want a command line utility, I really don't want to install npm then mess with any problems with that. I want to simply 'brew install' everything like that and be done.

Sure, if I need the source of something for compilation or testing, i.e. 'pip install numpy', then having that handled by pip + virtualenv makes sense.

Besides that use case, brew should handle everything to reduce conflicts.

What would you say is the difference then, between Maven / Gradle and say, NPM? I haven't used JVM tools, but what you are describing sounds like how NPM works. You specify that your project needs library X, version 4.5, in your package.json file and then NPM handles the rest.

Used both, difference is non-existant. Not sure what parent means.

It is similar, but you need to have npm installed first don't you? With gradle you typically interact with it through a shell script (aka gradle wrapper) and that downloads the correct version of gradle if it isn't already present in the project directory and then gradle downloads the dependencies. The person who sets up the project would generate the wrapper script using gradle.

> It is similar, but you need to have npm installed first don't you?

Same thing for gradle, you need to install it at first place. There is absolutely no difference except the language that is being used. And yes you can use shell scripts with npm too, except your shell scripts aren't going to run on windows.

Does npm have this?

https://docs.gradle.org/current/userguide/gradle_wrapper.htm...

I think that's what GP was referring to -- it's not a shell script for automating use of the tool, it's a shell script for automating acquisition and initialization of the tool.

When you fetch a package with npm, the package will either be : a library(installed locally) or a command line tool (installed globally). You can also run any number of tasks per project (npm run <task>), described in the npm manifest, or execute tasks before or after installing, and before publishing a library.

Right, but that doesn't address how you acquire/install/configure npm itself, which is one of the main benefits of having gradlew[.bat].

You need to have node installed already. Once it's installed you'll have npm because it's bundled with node.

I googled quickly for gradlew.bat and from the first link I opened it looks like it depends on java being installed already.

Node already includes npm because it's the official package manager. Java doesn't include gradle, presumably because they have multiple options and none of them is considered the official one?

I wasn't aware of the implicit "if you have node you have npm" relationship -- thanks for clearing that up.

For one, you can't install system-level dependencies with Maven. `gradle install <foo>` simply isn't a concept. Dependencies you download for a project are confined to that project, and optionally cached for other projects dependent on the same versioned artifacts.

The end result is that there is no "global" dependency like you would install with `npm install <foo>`. There is no dealing with dependency mismatches between projects that depend on "<foo> >= 1.2" for example, which can be a hassle to track down in the "global dependency installation" model most often used by these language-level package managers.

IMO this, the Maven and Gradle way is the way it should be. Global dependencies are a nightmare.

What you're describing for Maven is exactly how NPM works. You run `npm install foo`, it installs foo locally, for the project you are working on. Again, I am having a hard time understanding what people think is different between Maven and NPM.

The language package manager can have so much richer information than the system one, is the thing. The system package manager only understands a crude, lowest-common-denominator notion of dependencies, and everything goes via the system repo which is never going to be remotely up-to-date with the language package repo. Finally most system package managers won't let you have multiple versions of a library installed at all, which makes using more than 2 programs at the same time virtually impossible (because they usually want different versions of some dependent library or other).

>The language package manager can have so much richer information than the system one, is the thing.

This is simply not true. Language package managers cannot describe the entire dependency graph of a piece of software because they can't handle things that aren't written in that language. For example, there are plenty of Ruby gems that require a C compiler and C shared libraries in order to work, but those dependencies cannot be encoded in that system. System package managers can describe the entire graph, thus providing richer information.

>Finally most system package managers won't let you have multiple versions of a library installed at all, which makes using more than 2 programs at the same time virtually impossible (because they usually want different versions of some dependent library or other).

Yes, this is a big issue with the status quo system package managers. Fortunately, there's hope on the horizon in the form of functional package management. Functional package managers such as GNU Guix and Nix allow for an arbitrary number of variants of the same software to coexist without conflict. Additional benefits include unprivileged package management, transactional upgrades and roll backs, reproducible builds, and no central point of trust for binaries (anything can be built from source).

Language package managers are good for quickly sharing code written purely in that language with other developers, but unacceptable for anything else.

> Language package managers are good for quickly sharing code written purely in that language with other developers, but unacceptable for anything else.

Right, so, what almost all of the FLOSS world is doing, almost all of the time. Miiiight be worth optimizing for that case. Especially if a proper PDM would compose well with larger systems...which it would!

> Right, so, what almost all of the FLOSS world is doing, almost all of the time.

Citation needed. I have been packaging (scientific) software for years and it's usually a happy mix of different languages and libraries.

> This is simply not true. Language package managers cannot describe the entire dependency graph of a piece of software because they can't handle things that aren't written in that language. For example, there are plenty of Ruby gems that require a C compiler and C shared libraries in order to work, but those dependencies cannot be encoded in that system. System package managers can describe the entire graph, thus providing richer information.

On the contrary, the system package manager has to have a simpler model, precisely because it has to handle multiple languages. A generic model is never going to be able to capture the ruby-specific details as well as a ruby-specific model.

Based on my experience, this also isn't true. I added support for Ruby gems in the GNU Guix package manager. The way it was done was by creating a Ruby-specific build system that Ruby-based software packages elect to use. We have a generic model that allows for many types of specific build systems to exist that deal with the domain-specific concerns. This generic models works better than a Ruby-specific model because, unlike RubyGems, Guix can capture the entire dependency graph.

Take the Pg gem, for example. In order to build and use it, you need the libpq shared library. RubyGems can't encode this dependency, but Guix can. The recipe for ruby-pg is only 25 lines long: http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/...

There was a talk given at FOSDEM about foreign packages in Guix with a focus on Ruby which is worth watching or skimming the slides: https://fosdem.org/2016/schedule/event/guixmodules/

Got to agree with davexunit here. Language package managers aren't going to handle your non-<language> dependencies. One place where this comes up in spades is with numerical software. Example: frequently you want to build numpy/scipy with fast math libraries (BLAS, mkl, etc.), which might be in C, Fortran or whatever. Python's myriad custom package managers have no support for that level of customization so you're back to manual installs.

Spack [1] attempts to solve this by allowing extensions [2,3] for languages that have their own module systems. Basically you can embed language-specific module enablement logic in the package, and you can enable/disable packages in particular Python installs. You can then build several versions of numpy with different dependency libraries and swap them in and out. It's a compromise between language and system package managers. Right now it's too stateful in that the modules are linked/unlinked into a python installation. We have plans to provide something more like virtualenv or conda environments.

Functional package managers are nice, as are reproducible builds, but they come up short for customizability. I'd have to write an awful lot of packages with Nix and Guix to try out many different versions, compilers, dependencies, etc. Spack has a syntax [4] that allows these to be composed arbitrarily. It's not fully reproducible right now (though it could be if we went down to glibc and used a chroot jail), but the packages are much more composable than the current crop of functional package managers.

[1] http://github.com/llnl/spack/

[2] http://software.llnl.gov/spack/basic_usage.html#extensions-p...

[3] http://software.llnl.gov/spack/packaging_guide.html#extensio...

[4] http://software.llnl.gov/spack/basic_usage.html#specs-depend...

> Functional package managers are nice, as are reproducible builds, but they come up short for customizability. I'd have to write an awful lot of packages with Nix and Guix to try out many different versions, compilers, dependencies, etc.

I disagree. At least with Guix you have programmatic access to the dependency graph (since package objects are just Scheme values). You can traverse the package graph and easily swap out all occurrences of a particular package. You do not have to write the package expressions yourself as you can just modify the package objects directly.

See also the paper "Reproducible and User-Controlled Software Environments in HPC with Guix"[1].

Since the paper was published Guix also gained command-line syntax to rewrite the dependency graph on the fly, e.g.

    guix build guix --with-input=guile=guile-next

which maps the "guile" input to "guile-next" when building the "guix" package. Recent versions contain a pretty generic package transformation framework.

[1] https://hal.inria.fr/hal-01161771/en

In Spack this would look like:

    spack install guis ^guile@<new version>

How do you handle optional dependencies in this model? Or, a more complex case, dependencies that are only present for certain versions of the package? Look at the gcc package in Spack:

    https://github.com/LLNL/spack/blob/develop/var/spack/repos/builtin/packages/gcc/package.py#L54

This builds gcc 4.x and 5.x versions, and it's pretty concise. The `mpc` and `isl` dependencies are only present when the version of gcc is 4.5 or higher, or 5.0 or higher, respectively.

There's an iterative DAG-building step called concretization that looks at these types of constraints and tries to solve for a satisfactory DAG, which is then built in a functional style. I don't see how I would model that in Guix, and but in Spack it's a few lines, and there's command-line syntax to customize versions on the command line, not just programmatically. e.g.:

    spack install gcc@5.0 ^isl@0.11.1 ^mpc@1.0.2

One more reason for me to enjoy Gobolinux i guess.

The people in charge there have either implemented or plan to implement support for those language specific managers into the Gobolinux tree.

As best i understand it, this by making the Compile tool able to use these managers to download, and then capture the output and place it into the proper place in the /Programs tree.