Babel – Nimrod's package manager
picheta.me
picheta.me
It would be really cool if languages just started using a common package manager instead. It could also be more fully-featured, able to do things like track dependencies that aren't written in the language like C libraries. We would also avoid having a ton of custom file formats and dependency solvers.
From what I've seen, Nix[1] is the perfect candidate for this. It's self-contained, can install packages locally and does a good job of simultaneously managing multiple versions of the same package: all very useful for development. It's also portable, working on Linux, OS X, FreeBSD and even Windows with CygWin.
In a sense, I suppose this would be like reinventing the system package manager. But I don't think that's a problem. Unfortunately, most Linux package managers seem configured in a way that is not convenient for development (ie global packages only). It's also a bit of a pain for a language community to maintain a package for everyone's favorite package manager; if they just settled on one (like Nix), they could have a single Nix repository similar to how they have a custom package list now.
A single package, for a single package manager that runs everywhere and is tightly integrated with the language, is the only reasonable solution.
Basically, I do agree on using a single package manager. I just don't think that the need to be integrated with the language is severe enough to warrant writing a new one for each language! It's much less effort for everyone involved to set integration up for an existing package manager instead.
In other words: replace Pip/Cabal/Babel/whatever with Nix, maybe with some plugins for the needs of your specific language. You would even still ship it with your language (optionally), so the only change would be maintaining a plugin, if necessary, instead of a whole manager.
So now, to use packages for programming language X, I also have to install Nix, be running a Unix-based OS (sorry Windows folks!), and have whatever other dependencies and toolchain Nix requires installed.
This is the core problem. A programming language needs a package manager today. Code reuse is too important for a language to succeed with out a nice user experience to get and share code and deal with transitive dependencies.
That puts a package manager on the critical path for a language. Given that, it's important that the package manager itself have minimal dependencies. It would suck, for example, if a package manager for Python was written in Ruby and required Ruby to be installed!
The only thing most languages' package managers are willing to depend on is the language itself and its core libraries. That means for every programming language, you get a new package manager.
It's not ideal, but it seems to be the best solution we've found given those constraints.
For example, rubygems cannot properly express dependencies on C libraries. Rubygems that include C extensions run the compilation step during "gem install" [1] (which typically assumes the presence of a C compiler, etc.). A gem like libxml-ruby assumes the presence of libxml development headers. (Nokogiri punted and includes an entire libxml source tree in its gem which leads to its own problems.)
Nix has some means to convert rubygems into nix packages, and it has a list of overlay dependencies to apply to various common gems (e.g. libxml-ruby gets enhanced with that dependency on libxml).
Language-specific package managers handle simple cases well, but other cases make increasingly large assumptions about the external world that complicate automation and confound repeatability.
[1] There is support for binary gems, but I don't know of anyone who uses these in practice, nor do I know of a repository of pre-compiled gems.
Nothing in this idea says you need to run a Unix based OS, only the two current implementations of the idea (Nix and Guix) do so currently. The concept can be applied to a language specific package manager (LSPM) too, and IMO would be much better than the way many of the existing LSPMs behave - which is to duplicate dependencies into project directories - almost completely eliminating the reason to have shared object dependencies over static linked dependencies in the first place - because you need to manually update the dependencies of every project when a dependency gets a bug-fix.
The real problem with LSPMs is that they're functionally incomplete PMs - they require you to (manually) use a system wide package manager to install the native dependencies which you inevitably need, because it's inefficient to rewrite everything in your language rather than provide bindings to a native library. The LSPMs basically spit out their dummy when you try to install something and a native dependency is not met - often giving a cryptic error message which doesn't explain the cause of a problem, or how to resolve it.
I've suggested several times what needs to be done to resolve this, and it doesn't involve throwing away the LSPMs, but augmenting them. The solution is to provide a system wide daemon which facilitates native installs required for by the language, whereby the LSPM can query and request the installation of packages over a well defined protocol, and the daemon forwards those requests to a system-level PM. This could also improve the state of GUIs for package management - allowing a unified interface to installing them rather than a dozen ad-hoc CLI interfaces.
Of course, I don't have all of the details on how this would be implemented - for starters, it would add a new dependency to every package manager using it (the IPC mechanism and protocol used), although almost any language can use a socket based IPC. Each LSPM would also need to introduce some new notation for "external" packages, which it doesn't manage internally, but instead sends off a query to the daemon, waits for a response which will either fail (the native package is not installed or won't be installed), which will also improve the state of error messages, or it may succeed, and the LSPM can continue building the rest of the project.
If such system is going to be built, the Nix model of immutable packages seems really the only sane approach to the problem. The identity-based approach to dependencies means that when building a package, you're always going to be building against dependencies the developers intended - because they're the same ones he/she tested against. Each time a developer builds successfully against a different dependency version he should add its identity to the package definition, confirming that it has been tested. If you decide you want to build against a different version, you can simply adjust the identities, but you should not expect it to work without some manual fixing - because people tend to break things. If you happen to have luck, and your modified dependency works - you should publish the updated package definition so other people can learn from your testing, and blame you if you published a broken package (ie, failure to build implies a bug, as it should)
That may be the right choice for the kinds of packages that Nix targets but not for other language's semantics. For example, I wrote the package manager for Dart and it specifically intentionally does use versions because forked shared dependencies don't play nice with the language's semantics.
> The solution is to provide a system wide daemon which facilitates native installs required for by the language, whereby the LSPM can query and request the installation of packages over a well defined protocol, and the daemon forwards those requests to a system-level PM.
That's a large ball of complexity and dependencies. It isn't clear that that's actually a more effective solution than just rolling a package manager for each language.
> The identity-based approach to dependencies means that when building a package, you're always going to be building against dependencies the developers intended - because they're the same ones he/she tested against.
That doesn't play very well with shared dependencies in many cases.
Ultimately, every package manager has a bunch of policies embedded in it. Things that it just decides are The Way Stuff Should Work. Those policies are heavily influenced by what how the programming language is used.
Trying to find the One True Package Semantics seems like a lost cause to me.
* developer X releases package A, version 1.0
* developer Y releases package B, version 1.0, depending on package A > 1.0
* developer X releases package A version 1.1 with breaking changes
* user Z with package A 1.1 installed attempts to build package B and fails
Who is to blame in this case? At first thought you'd blame developer X for releasing breaking changes without bumping up the major version, however, there was nothing in his "contract" that says he cannot do this - moreover, even if one specified up front that they will "not release breaking changes without bumping the major version", how can you even enforce that at the software level?
The way we actually model the "blame" in the industry is to blame Z - the idiot user who can't build software, who failed to follow incorrect documentation and feels like an idiot because he doesn't know what was wrong - then he wastes significant time searching for solutions, debugging build scripts, re-versioning the dependencies by introducing A10 and A11 in place of A, because he needs both, and his PM can't have two of the same name, and the names can't contain periods (Arbitrary restriction of some PMs).
The actual blame here lies in Y - he is the one who documented that his software will work with any version of A greater than 1.0, but I find this declaration to be idiotic - he cannot possibly uphold this "contract", because he is not in control of A. The best guarantee he can give is "I've tested this with 1.0, but it may work with any later version, however, do so at your own risk."
Ok, so there's no really a "contract" to be held - particularly for public domain works and "permissive" licensed works where the developer is just giving stuff away for free, but in the case of copyleft works - such as the GPL, it specifically requires that you distribute all of the scripts required to build the software. While perhaps not a violation of the license in the legal sense - releasing build scripts which simply don't work is a violation of the spirit of the GPL.
We could argue that "may work with any later version" is precisely what we mean with "> 1.0", but it confuses me as to why anyone would think releasing an untested build script would ever work - you would never release software into production without first testing it. Releasing a package with a dependency that doesn't yet exist cannot possibly be tested - it's not fit for production. If a developer has tested a piece of software with all dependency versions between 1.0 and 2.0, and decides they all work successfully, then I see this as the only real case where ranged versions are acceptable - except in this case you have a set of exact dependencies tested against which could be used instead of the range, so why not specify dependencies as a set?
These complaints are not specific to any language semantics - it's just a case of untested code being unreliable, and tested code, where all the "tests" are encoded into the definition of the software (thereby removing the "hidden knowledge" that goes into building it), is close to perfectly reliable - any failure to reproduce was probably beyond the control of a package publisher anyway (although when we have ReproducibleBuilds, this should never happen.)
> That's a large ball of complexity and dependencies. It isn't clear that that's actually a more effective solution than just rolling a package manager for each language.
As I stated, LSPMs are an incomplete solution - they don't provide the complete "dependency resolution" which we expect from a PM. If we take for example, I release a Cabal package for Cairo bindings in Haskell, and I run "cabal install Cairo", what actually happens?
If I'm lucky, a compatible (by magic) version of native cairo is installed, and it builds fine. If Cairo is not installed, it will fail with some cryptic message saying ExitFailure 1, something or other about gtkhs-buildtools. If I'm an expert already I can wave my magic wand and fix the problem, otherwise I'm googling a solution. Turns out the problem is simple in this case, I don't have the package "cairo-dev" installed on my OS, and I need to install it manually through another PM, say, apt.
But then this defeats the purpose - the LSPM, Cabal, has failed at dependency resolution - it can't resolve Cairo because it's beyond its scope - having Cairo on the system is the responsibility of the system PM (which is really an LSPM where the specific language is C, etc).
The point here is to eliminate the need for manual installation of cairo - by providing the means for Cabal and apt to talk - Cabal can simply ask "do you have cairo", to which it can reply "yes", or "no, would you like me to install it", or "no, go away".
This may not be such a concern in dart, since you don't really have bindings to native libraries.
I don't see how this is a large ball of complexity, but I think it could reduce it. As things stand, some LSPMs attempt to somehow resolve native dependencies through ad-hoc means, by looking for them in specific locations in the filesystem, or by dumping the native dependencies into the project folder itself (good luck picking up bug-fixes and making portable projects this way). Other LSPMs simply shut down and cry when you need a native dependency, "it's not my problem".
Rather than trying bridge individual PMs on an ad-hoc basis, it would make more sense to come up with the right abstraction/verbs for them to talk - and each one have an adapter to a global "package manager manager daemon",. This would be like "QUERY cairo", "INSTALLED", "NOT INSTALLED", etc. (As I said, I don't have all of the details, as that would require a large social effort.)
The reason the Nix model would be required for this kind of collaboration of PMs should be obvious - if I query "cairo", is it going to ask apt to install cairo, or ask Cabal to install cairo? It can't possibly guess without the user disambiguating - adding version numbers here can't help either. The only solution here is to make sure there can be no ambiguity between them - different names won't work because the different package managers have distinct repositories managed by separate communities. The solution is an identity, hash-based, or based on a cryptographic signature - it's required for any kind of security if we were to have PMs talk to each other - however, using hash based identities, we can treat the knowledge of the hash as a capability, and if two different package managers (such as apt and yum) provided the same package identity, it wouldn't matter which one performed the installation, because it's the same capability.
I'm not trying to find the "One True Package Semantics", but rather, create an environment where each PM can handle its own niche, but bridge the gap that's missing where they simply stop working - by having some kind of shared language between them.
It would be helpful of every successful package DAG for that occurrence we communicated somewhere for others to use.
If we're looking for a way to "communicate" working DAGs for other people to use, I think the best place to look is existing in DVCSs - submodules/subrepositories basically provide what we need - because every commit to a repository using submodules refers to exact commits in exact branches of the submodules. If we managed to use git ubiquitously in such way that all of our dependencies were submodules, and in turn, those dependencies used submodules for their own dependencies, we have the entire DAG that would otherwise need to be written as Nixpkgs.
This also solves the version explosion issue, where there could exist dozens or hundreds of derivations of the same packages with slightly different configurations, bug-fixes or other changes - because a "derivation" just becomes a specific commit in a specific branch of a repository at a known URL, and to utilize the different version from a dependent software, you just update the submodule and point to the desired head.
Well yeah if your product is uni-language. If it is not then you are installing python pips, ruby gems, java jars, make && make installs for C++ code. And some stuff is uni-language, granted. A lot of Github demo projects and libraries are uni-langue And then they might have cross-dependencies, some have transitive dependencies. And you might have to uninstall some old package and install new one, except you can't make make & make install just dumps files on the file system and doesn't know how to properly clean up after itself.
Then in many cases uni-platform is common. That calls for using the OS package manager. Say if your services sit on tops of CentOS 6.5 then why run around sprinkling files all over the file systems via shell scripts or re-implement an adhoc package manager in bash when you can just use RPMs.
I have seen this all too many time "RPM writers are stupid, look at this complexity, I'll do it quicker". In the end they come up with something messier, less reliable, not-quite-working-right re-implementation of RPMs.
On the client side, nix expressions' dependency resolution is name-based. Meaning you can say: my program depends on "ruby" and the definition of "ruby" gives you the version of ruby. As program dependencies tend to evolve quite a bit it would be a chore to update all the dependencies manually, we are used to having a sat solver to do the work for us.
Both things could be bolted onto Nix I think. You just need that publishing infrastructure (github is an existing one for a start) and a dependency solver that generates nix expressions.
I have some code that uses Bundler to convert its Gemfile.lock (resolved dependencies) into a set of nix expressions. I think it's a nice way to go.
(Though Bundler is aimed at end users, which for my use case means an annoyingly large amount of functionality and no stable api; e.g. it insists on installing gems as part of the dependency resolution process.)
Last I checked, Nix has no way to publish just a small nix expression for one project. You'd really need to include the entire stack of nix expressions for all your dependencies, all the way down to gcc (which is exactly what nixpkgs is), or somehow declare "depends on http://nixpkgs/2014-07-24.tar.xz".
It's definitely possible to do that and even link to nixpkgs but then your projects dependencies might break each time you updated nixpkgs. I don't know of a nice way to solve that.
My sample looked something like that:
let
pkgs = import <nixpkgs> {};
stdenv = pkgs.stdenv;
in
stdenv.mkDerivation {
name = "my_project";
src = ./.;
...
"The Purely Functional Package Manager" - what does that mean and why do I want that instead of ruby gems, php composer, jar files, maven, leinegen, nugget, etc?
"Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible." - What if I want Windows? (serious question)
I could go on, but none of what is on the landing page explains why Nix should replace ruby gems, php composer, or any other package manager. It's like saying you are mad at the 13 other standards so you create the 14th standard for people to standardize around. Now you have 14 different standards.
The reason we want referential transparency for packages is to make them reliable - so that another person building your package gets the same result you did, and something like an environment variable, missing packages, or different native package on his system doesn't cause the software to break, or simply fail to build. If you've ever tried to build some software and configure or make has returned some cryptic error you end up googling - this is exactly what Nix solves.
On "What about Windows?" It's a serious question, but the serious answer is that nobody is interested - if Windows users really wanted it, they'd build it, and so far, they're two decades behind even the most basic kind of package management offered by Debian. The best Windows has so far is nuget, which is language specific, and also pretty miserable at actually solving the package management problem - it's a convenient hack.
A more interesting project on Windows is CoApp, although I'm not sure if it's still in development. The idea behind CoApp was to create a Visual Studio project file for common open source software and test them - such that Windows users will not be required to install mingw/msys/cygwin to provide some basic functions to their machines.
Nobody is suggesting that Nix should replace all the package managers, but the idea behind it is probably more valuable than others going into the future where complexity will continue to increase, and traditional package managers have stagnated in solving our dependency problems.
The issue of standards and "one to rule them all" isn't really what nix aims for either - it's in fact, completely incompatible with other PMs by nature - because it requires that dependencies are exact, and other PMs simply cannot provide anything of that nature - they only produce "approximate" packages.
I personally am interested. By what you and a lot of people here said Nix's Windows support is not great (and no, support through Cygwin does not count). So Nix is already ruled out for me.
I am willing to bet that getting Nix to work reliably on Windows would have taken far longer than Babel.
Trying to make Nix work on windows would require making up "psuedo-packages" for components of windows, where some hash of the binaries is used to identify them - one problem is we can't guarantee that components in windows can even have an identity which is common between installs - there could be a significant amount of unique configuration embedded into each install of the OS, which would give unique hashes, making it ineffective to even bother trying to make package trees where dependencies are exact - every user would need to maintain their own tree.
If anything like Nix were to come into windows, it would probably need to come out of Microsoft to be effective - they'd need to acknowledge the merits of Nix and decide to push updates through a similar model.
CoApp was of interest because its author was in Microsoft, and had potential to make an impact on the way software is delivered on Windows - it used separate shared libraries for each dependency without dumping them into project folders (by making effective use of the SxS technology in windows). It also required packages to be signed, which would've provided some of what is necessary in a model like Nix's, and in CoApp, each package is maintained in git, in a separate repository from the original projects. It looks like CoApp is still being worked on, although there's little activity - Nuget basically pushed this project to be stillborn, because Nuget provided an immediately convenient solution to the woes of Windows developers, without actually solving package management - the rest of Microsoft jumped on it and CoApp was ignored.
Perhaps the downside of CoApp is the requirement to support Visual Studio, and to not acknowledge the existence of mingw/msys, which are really a requirement for building many projects, and the effort to port these builds into VS is non-trivial. I agree that cygwin is not a solution, it shouldn't be considered as such as it's more like the windows equivalent of WINE. Mingw/msys suffers from nearly all the same woes as the rest of windows, although it now has it's own "PM" (mingw-get), which is an improvement over the way things were done before then, mingw-get isn't a very sophisticated PM, more like a specialized downloader.
This goes against what i just said above, but I think any potential solution on Windows needs to be liberated from Microsoft (and MS specific toolchains) - that is, a solution built around mingw and msys, whereby the mingw base packages become the bootstrap packages for an immutable model, and where those packages can reference Windows components with some alternative means to give them identities other than hashing of binaries - for example, the PM could provide some way to override the requirement of hashes for dependencies, granting that special permission only to mingw packages. The rest of user-land software would be built with mingw.
Of course there is conflict here on building with VS versus mingw - the binaries produced by each aren't really compatible at all, so the use of both would require a dual build system. There's something out there like this - the KDEwin project, which has packages for both toolchains - however, there is still significant work needed to improve the state of KDE on windows, and the PM itself.
Nobody is really suggesting that we don't use Babel, or that Nix is going to replace it overnight - you're absolutely right that getting something like Nix on Windows will take monumental effort, but we're sharing ideas for more long-term solutions to packaging problems, which continue to get more complex, and tools like Babel don't actually do much to solve that growing complexity.
But a package manager that relies on Cygwin certainly isn't going to win. Most language-specific package managers are already doing better than that.
More specifically, the import/require statements in a language, coupled with the information you can get from a package manager and syntax tree parsers provides enough information to map out entire programming ecosystems.
If there were ever one package manager to rule them all, it would have to support bindings (possibly as a plugin) for every language for it to truly be as valuable as the language/runtimes it supports.
Furthermore, writing a package manager is usually one of the first large scale projects common to most language ecosystems that really exercise what a language is capable of. Other tools for working with the language, such as parsers, interpreters and compilers are among the first projects in any ecosystem.
Just makes it harder to find either.
Maybe it was confusing already?