Which programming language has the best package manager?
blog.versioneye.com
blog.versioneye.com
Ruby's "gems" are source packages. When a gem is installed, arbitrary code is executed, C stubs (or even C libraries!) are compiled, and the resulting artifacts can be very different on different systems. Some Ruby gems even modify the Ruby code at install-time. The same is true for Perl (CPAN) and Python (setuptools, pip, etc)
Java and Clojure packages are artifacts. Each jar is a finished, built product. If it has JNI stubs, those stubs were pre-built against a specific version. The artifact is identical on every installed system. No arbitrary code is executed. Reasoning about the contents of a Clojure package does not require me to solve the halting problem.
The artifact approach is unambiguously better for any production deployment. The source-based approach found in Ruby, Perl, and Python is a problem for me more often than a solution. It is hypothetically great that I can use the same "gem" on OSX and Linux, but it is more important to me that I get consistent deployments on two different Linux systems.
Which is precisely why many Linux distributions use them, and "app stores" were already a thing in Debian in the mid-90s. Every time I read about a programming language's "package manager" I can't help but feel that the creators of such are reinventing the wheel, poorly, and mostly to paper over issues in backwards OSes that don't have proper package management.
Of course they could target Nix instead, but how many people use Nix?
More than use the new system they developed instead, at the outset.
1. Allows developers to distribute their stuff to users on those OSes. 2. Requires the least amount of effort as possible on part of the developer. 3. Does not require the developer to custom-package their stuff multiple times, once for each OS.
Face it, developers are lazy (not meant in a derogatory way). Most solutions do not meet requirements 2 and 3, which is why developers will largely choose to avoid them until they have no choice.
And also, what matthewmacleod said in this thread is very true.
To take Ruby as an example - I have lots of different interpreter versions installed. 1.8.7, 1.9.3, 2.0, 2.1, jruby, rubinius. There's some scope for managing this in OS package managers, but it's already complex. Then for each of these environments, I want a different set of packages with different versions (some are compatible with older versions, and some aren't, for example). And finally, I want different versions and combinations of these packages for different apps that I might want to run - again, there's limited scope to implement this in an OS package manager.
The big difference between dpkg and gem is that gem allows you to install more than one version of a gem at a time. With dpkg you overcome this by creating a whole new package when the major version changes. Thus I can install libcap and libcap2 simultaneously. Any breakages in minor version changes are considered significant problems.
This is the right thing to do in a production environment: minor & patch level increments are very likely bug fixes and security fixes, and should be applied to all packages that use that library.
But it's a PITA for development. And since package manager developers are mostly developers, they did what was right for them, and allowed multiple versions to be installed simultaneously.
Today, the right answer is to use something like Vagrant or Docker so that development environments are much closer to production environments.
1. RHEL / CentOS solve this problem with "software collections," aka "scl." Using scl gets you the advantages of ye olde retargetable RPMs but without all the hassle and suffering. Makes it relatively easy to scope your packages that way.
2. fpm doesn't do nearly as bad a job as it used to. Given a gem, it can correctly build it, assign its C dependencies, and assign a prefix so your gems for ruby-ee don't overlap with ruby19 or ruby18.
It's kinda funny how they named the latest and greatest in the Python world because of this: http://wheel.readthedocs.org/en/latest/
I bet that's doable ...
Change string.c, recompile, if a gem can execute arbitray code.
There are surely many others, these are just the ones I have at hand as examples.
Edit: Link to PEP for Wheel: http://www.python.org/dev/peps/pep-0427/
Python libraries that use setuptools are the same way.
My point is: a source-based approach is not the tool to blame for consistency, it's used with lazy concern, and that's a mistake onto the crafter.
I work on two different platforms as one. Virtualization achieves that.
Relying on virtualization for repeatable source-based deployment is like screwing in a lightbulb by rotating the entire building around the bulb.
Two approaches you can take:
1. bundle package will package up all of your installed gems, which you can then deploy with your project.
2. Your build server can install all of your gems, and then you can package up the resulting directory and deploy it - you could also build e.g. an rpm package.
Installing all of the gems then packaging the resulting directory is an example of an artifact-based approach.
I prefer to do it gem-by-gem using 'fpm' in order to maintain better dependency data -- fpm will invoke rpmbuild with the right options to find shared objects and automatically attach correct dependency data for the ruby runtime, C libraries etc
But with Node.js, the typical deployment scenario for us, and I believe others as well, is to simply tar up the entire app including dependencies.
I'm not sure if this is possible with the other scripting languages you mention.
I also don't believe this is as big an issue in development for Node.js, because most packages are pure-JS, and don't run any build steps whatsoever.
Finally, I think it's also very convenient to have my dependencies' sources available in development. I trace into them quite often.
Which is only a problem if you're still doing application deployments directly to production machines, which is becoming less and less common. Tools like Docker, Packer and Aminator allow you to do the package install at build time.
* centrally hosted
* mirrored all over the world
* packages are automatically checksummed & the checksum is compared on install
* is is quite easy to publish packages
and very important in my opinion
* it encourages you to write tests. Unit tests included in packages are automatically executed when it is installed. Many systems (semi)-automatically report back when the installation of a package fails. Hence you get tests on tons of platforms for your package
* pod documentation is automatically available on cpan.
Not only will you get comprehensive tests and reports of all the packages with different OS/language versions, it is a great test of the language implementation with different C compilers etc.
dzil new My::Module
<build library>
dzil test #Make sure the build and tests go okay
<set version and add changes to changefile>
dzil publishI think npm's approach is pretty cool -- it sacrifices disk space for portability (package dependencies are always installed in the node_packages directory of the package root), so while you might effectively have 3 or 4 of the same coffee-script folder in your install, you don't have to worry about virtual environments for development and you don't have to worry about dependency upgrades breaking some other package. I think the nature of javascript allows this to be effective considering that many dependencies are smallish snippets of code.
NPM is doing a pretty good job. If they would sign their packages and make the licenses mandatory it would be the clear winner. Only 2 points are missing.
I don't know that I agree that having the dependency definition in native code is a source of vulnerabilities; it could make the definition easier to write for people packaging code in that language. Also, I think JSON could be considered native code for npm since it is written in nodejs/javascript.
Usually but not always. You can also have peer dependencies: http://blog.nodejs.org/2013/02/07/peer-dependencies/
Unfortunately, Nobody Cares About Signed Gems: http://www.rubygems-openpgp-ca.org/blog/nobody-cares-about-s...
We've tried making a difference in Phusion Passenger by setting an example, and supporting gem signing (http://www.modrails.com/documentation/Users%20guide%20Nginx....). All Phusion Passenger gem releases since 4.0.0 RC 4 (1 year ago) are signed. All our other gems (default_value_for, etc) are also signed. Unfortunately not many people followed.
We'll continue to sign all our stuff, but it's sad that it never took off community-wide.
If we want to have more security in the Ruby community then there is only one way. RubyGems has to decline every unsigned Gem. Signing Gems must be mandatory.
I find the following cons with NPM:
- shrinkwrap does not work as well as the alternatives in Bundler (yes, that's plural).
- specifying versions as git hashes does not verify the hash on "npm install"
- npm link is more awkward than using :path => in Bundler.
The big difference between NPM and bundler is that NPM allows common dependencies of dependencies to have different versions. This is usually awesome, except when it isn't -- it can cause subtle breakages if you try to pass an object from one dependency to another. This is very rare though, and if I had to choose, I'd choose the NPM behaviour.
- Allowing multiple versions of the same package in one application is asking for Cthulhu to climb out of the ocean and burn a whole in your mind and your program.
It's virtually trivial to set up a gem server, for instance, but I wouldn't know where to start for the others.
Every one of these tools is 100% focused on supporting a single language. They go so far as to use that language as the build spec so even attempting to bend it to another platform feels alien. It's okay if I only ever want to use Ruby or Javascript will be my thing for the next decade. But what if I want to take my existing npm or gem workflow or knowledge and apply it to a new language? I have to start all over from scratch. I'll have no idea how to fetch dependencies or package up my stuff for sharing.
If I start to tinker with more than a handful of languages, my head is going to be full of details on package management rather than APIs or library capabilities. Who wants that?
PIP is a good tool. But in my opinion the definition of dependencies shouldn't be in setup.py file. Because it's too easy to execute random code in a setup.py. A JSON file is the better choice for a project file, in my opinion.
When you type 'pip install package_name', pip connects to PyPI to retrieve package_name.
In fact, installing the entire Haxe toolchain is the easiest I've ever experienced.
There are also solutions such as https://github.com/pusewicz/rubygems-proxy allowing you to cache / proxy to rubygems.org
There is also http://gems.gzruby.org/ if you happen to be behind the Great Firewall of China
Gradle is another build system with dependency management.
The initial setup of publishing an artifact is high on Maven Central, but Sonatype makes it easier. Even easier is publishing to clojars.
But I still think that the initial publishing on Maven Central is more difficult then it should be. Pretty much every other package is doing a better Job on publishing. It took me less then a minute to publish my very first Ruby Gem. But it took me 1 week to publish my very first Jar file on the Maven Central.
Clojars is doing better.
Sent from my Apple MAC.
Or would we consider apt-install <whatever>-dev to be that?
Personally I've always found Maven to be a pain, but a necessary evil for larger projects / teams. I've tried using it for smaller personal projects and the benefit wasn't worth the time invested.
I'd rather not invest more time learning Ivy / Gradle / anything else until someone can convince me it's not the same old crap under a different label...
The Gradle overview at http://www.gradle.org/overview says "The Gradle design is well-suited for creating another build script engine in JRuby or Jython. It just doesn't have the highest priority for us at the moment."
Perhaps if Gradle easily enabled any JVM language to be used, there'd be far greater uptake. There's a few of us out here who don't like using Groovy.