Ruby on Guix | Hacker News Reader

Ruby on Guix

dthompson.us

74 pointsrekado49 comments

I completely see the benefits of using guix, transactional package management seems at the very least an interesting alternative to docker in some scenarios, and could be very interesting from a tool automation perspective.

I don't want to sound negative, but I think the biggest problem with getting me or other ruby developers to package our gems to guix would be the following. The bundler/rbenv system I've been using for years is pretty darn good, personally I haven't had problems with it. I've used it in massive and small projects, and admittedly the only inconvenience is having to manually or through orchestration install the dependencies of certain gems, of which there are also only a few (mysql2 with libmysqlclient-dev and his nokogiri mention are probably the most notable). He does point this out but if you're using docker or some automation tool this is a pretty minor point.

Also I already ensure my gems run their tests on mri, jruby, rubinius, etc. and this seems like extra work for little benefit, especially when most in house deployment scenarios are running with docker or chef/puppet on top of the classic rbenv/rvm and bundler setup. My biggest question is after upgrading a gem version how easy is it for me to push the new version of the package to guix?

So essentially, I wish this article had more arguments to show why guix will better the life of ruby devs.

I don't know if it's the same with Guix but in NixOS it's just a matter of running `bundix` and all the Gemfile.lock dependencies are converted to a nix expression. Gem developers can keep developing as usual.

Maybe the only restriction is to make sure the gem has configurable paths if it tries to access system files.

I'm experimenting with it at the moment and converted my blog to be managed by nix on OSX. It's not a walk in the park but it's working. Clone the repo ( https://github.com/zimbatm/zimbatm.com ) and type `nix-shell` (needs to be installed first) and you should get all the blog dependencies available in your environment.

> I don't know if it's the same with Guix but in NixOS it's just a matter of running `bundix` and all the Gemfile.lock dependencies are converted to a nix expression.

I'm glad to hear that the infrastructure is working for you. There are, admittedly, some kinks on master, though I have a PR to make things much, much more robust:

https://github.com/NixOS/nixpkgs/pull/8612

Unfortunately, while Bundler is generally a great, useful piece of software, there are aspects of its design that make it very difficult - borderline antithetical - to cooperate with packaging; if you look at the commit message in the above PR, you can see a thorough description of the challenges.

I have an open PR for Bundler to try to make things a bit more amenable, and recently got the thumbs up with respect to the general idea:

https://github.com/bundler/bundler/pull/3775

Unfortunately, most Ruby deployments are quite imperative in nature - just push to a git repository, bundle install, and then bundle exec (that's leaving out all of the subtle prereqs: `apt-get install ...`, etc), as with Capistrano. There doesn't appear to be as much of a culture around packaging (yet), and so there hasn't been a demand for the tools to meet those needs.

I hope that we can push through those problems and make it trivial for people to enjoy the ease/dependability/resiliency of package management, as many people have no idea what they're missing out on.

Nice ! That should solve the issue where all gems are re-installed whenever the Gemfile.lock is changed.

I agree that bundler is mainly focusing on the user interactions and is good at that but never look inside. I'm not even sure if the dependency resolution mechanism can be used as a library.

In my opinion bundler should only be used as a dependency-resolution manager. Once gems are installed then they should use good-ol' $LOAD_PATH. Unfortunately there are the git gems and also rails which insists on using bundler.

EDIT: got any thought on adding system dependencies per gem ?

As someone who deploys Ruby software to our servers, I've come up with a formula that's served us well.

We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm.

In other words, the packages are completely self-contained; no gems are used from the environment.

The key to this is the --path option to bundle install, i.e.:

bundle install --path vendor/bundle

This command makes sure all my gems are in a sub directory of my project called vendor/bundle. When creating the debian package using fpm which includes all the source code, vendor/bundle also comes along with the package.

After installing the app on our Linux servers, 'bundle exec rails server' uses the gems from vendor/bundle.

This approach does require that I create the debian package on a machine that closely mirrors the deployed servers, so that compiled gems make any sense.

Most all the issues described in the article have not been problems because of this approach. I used to use rvm as part of my build process; it was very brittle. I'm completely happy now; just wanted to share in case that helps someone else.

>We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm.

That is exactly what I've started doing at work, actually. It's a short-term win and rather simple, but it's a pretty big hack. Package bundling is quite poor form as it leads to file system duplication and for each duplication you have an additional location to patch when a security vulnerability is inevitably discovered. Ideally each Ruby gem would be its own package. I'm also concerned with reproducibility, so I build packages in as isolated of an environment as possible where I run 'bundle install' with a fresh vendor/bundle directory. Builds take longer than they should because of all the re-downloading. Using Guix, where I can easily represent each Ruby component in a package object, I can take advantage of its content-addressable storage to save time on repeated builds by using the cached gem builds from previous times.

Valid points.

We use a 'monorepo', so everytime we update any gem dependency, all of our apps get rebuilt entirely. So, updating all of our packages or not isn't a problem.

And I like the duplication, believe it or not. After having debugged environmental problems with dependencies in various languages over the years, I'm happiest knowing the dependencies in question are in vendor/bundle; end of story. Sometimes, in case of a nasty bug, I have a one-liner fix and I can go to vender/bundle, tweak the gem, and know I only affected the app using it. Those are things I prefer over some lost hard drive space.

Build times are a little long for us, though. That is true. I do a fresh bundle install on every build. That is the second longest part of the build, behind running our dog slow web tests.

The duplication is less of an issue in the corporate, one application per VM environment. However, my focus isn't solely on that. Web applications are really hard for most people to self-host. They are made a bit easier by things like OmniBus packages that bundle absolutely everything, which makes the user dependent on each application author to ship security fixes to software they didn't write.

There's also the issue of stateful package managers that can break in the middle of a package install and you're screwed. Most of the time that doesn't happen, but I've been bitten before. So the fpm approach is a great short-term win and I want to roll it out to the production systems at work soon, but in the long term I think we need functional package management to make our systems more resilient to failure.

Yep, this works well. Not sure if fpm handles native package dependencies too, but that's a very-nice-to-have.

Looks pretty cool, though it is a pretty daunting task to convince all gem developers to test/make their gems compatible.

Perhaps this could make http://devblog.avdi.org/2015/08/11/what-its-like-to-come-bac... easier ;)

I don't expect to convince all gem developers by any means, but it doesn't hurt to ask nicely if the brave few would give it a try. ;)

I certainly will :)

What is exactly wrong with piping curl into bash? I'm trusting RVM anyway. As for functional package managers, I'll believe it when I see it. The reason things are brittle is that any time something in the middle of a dependency chain changes the whole chain may break. The more you freeze things, the greater chance there is to freeze up a security hole (see what is going on with Docker). So you have two choices: be ok with the freeze, or update the whole chain. In this respect I prefer the latter, and I especially prefer the way Ruby does it where, due to the global namespace, I'm forced to upgrade the gem everywhere so I end up fixing things and keeping the whole project up to date. While this seems frustrating at first, once you realize that everyone in the Ruby community is doing it, you realize that the reason it is usually painless to update your gems is because other people have fixed those other holes. What is especially nice is that you only have to remember things that are broken at T0, unlike, say, Node, where a popular nested node_module may have 10 different versions in your dependencies and you keep hitting the same bug or security hole over and over again.

I've found all the hate surrounding this practice to be pretty misdirected. As long as it's served over https, it's functionally equivalent to "download and run this random package installer!", which is generally tolerated.

For the record, I do not tolerate "download and run this random package installer!" either, without GPG signed binaries and corresponding source.

I am quite sure RVM now offers a signed download too.

https://rvm.io/rvm/security

Can guix be used somewhat like virtualenv?

I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications, but I would very much play with it if it were possible to do something like this:

    guix virtualenv myapp
    cd myapp
    guix activate
    guix install postgres # installs a local postgres installation available only on this "environment"
    guix environment -l myapp.scm # install app dependencies
    <do work>
    guix deactivate

This would be a killer feature for me and I would ditch virtualenv, pip, npm, perhaps Docker in a heartbeat.

> I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications

That's the beauty of functional package management: no global state is modified. Whatever Guix does is restricted to its own namespace, e.g. the /gnu directory. Every package that is built or installed ends up there and is completely isolated from the rest of the system.

To actually use the software you use a link to a profile (which is located in the "local state" directory of Guix).

You can install packages into separate profiles with Guix:

    guix package -p /path/to/dev/profile -i postgresql

Or you can just run a shell in a volatile environment as specified in `myapp.scm`

    guix environment -l myapp.scm

To exit this environment, just exit the shell.

In summary: it can already be used just like you want, but it's much less verbose :)

Yes, the 'guix environment' tool is like a language-agnostic virtualenv. Guix does not use or interfere with any components from the host distro, so you can install Guix on top of Arch Linux without fear. We have a binary installation method[0] that isn't too difficult that you can try out. I use Guix on top of an Ubuntu host at work.

For your PostgreSQL example, you'd still need to do things like initialize the DB cluster and start the daemon on your own (the GuixSD distro has a system service for this[1]), but to spawn a shell with the postgres daemon and client programs available you would run:

    guix environment --ad-hoc postgresql

Exiting the sub-shell would do essentially what you had in mind with 'guix deactivate': Make postgresql available for garbage collection (via 'guix gc') if/when nothing else is using it.

[0] https://gnu.org/software/guix/manual/html_node/Binary-Instal...

[1] https://gnu.org/software/guix/manual/html_node/Database-Serv...

The point was to install the deps into my project tree, instead of putting everything into /gnu, but to think of it that's not actually much of a problem, it's just personal preference/bikeshedding.

I'm installing it and see how it goes, thanks.

"Installing" a package with Guix really just means that it is referenced through a link. /gnu is just a cache for "evaluated" package recipes. You could create a profile that contains your symlinks to the items in the store wherever you want, including the currenty directory.

    guix package -p $(pwd)/.guix-profile -i what ever you want

Programmes in this profile can then be executed like so:

    ./.guix-profile/bin/what

But that's not necessary when you could just use `guix environment` and enter a shell where everything is already hooked up.

nix has tools to convert gemfile into nix expressions http://lists.science.uu.nl/pipermail/nix-dev/2015-January/01...

Notice the striking similarities of package management definition files and configuration management, where the latter does more related to state, coordination, attributes and templating files. Packages are just more generic containers of resources and metadata, configuration management makes those resources concrete on specific systems and applies tweaks to converge on the desired catalog state.

Btw, has anyone written a usable Guile-based cfg mgmt tool yet?

Configuration management is slowly coming to Guix. Guix is the foundation of the Guix System Distribution, which is a system instantiated from a declarative system configuration file /etc/config.scm. There is work under way to extend this to managing multiple machines.

> Btw, has anyone written a usable Guile-based cfg mgmt tool yet?

When it comes, I'm certain it will come to Emacs first :)

LOL. Emacs is RMS'es CLI to the world. It's fine. ;)

>Btw, has anyone written a usable Guile-based cfg mgmt tool yet?

That would be Guix. We use the same primitive utilities for package management and system configuration management. I'm typing this from a laptop running the GuixSD distro, and if a system upgrade were to break things, I can roll-back to a previous, working generation of the system at boot time and keep on going.

That make a ton of sense because the boundaries between packaging and cfg management are artificial.

I like the approach of using gs (simple gemsets) [1] and dep (simple dependencies) [2]. I've been using this approach of handling dependencies for a while now.

I've written a simple shell script as a replacement for gs (since I constantly forget if I was in a sub-shell or not). All it does is change $GEM_PATH and $GEM_HOME when $PWD changes (while staying in the same shell). If the current $PWD has a directory ".gs", `gem install` will install them there instead of globally. All I have to remember is to create a .gs directory when I start a new project.

It does not address all issues highlighted in the article, nor do I know how well this would work in a production environment with automated deployments. It's probably more of a work around than a solution. I vastly prefer it for personal projects, though. Hate solving gem conflicts with "bundle exec".

[1]: https://github.com/soveran/gs [2]: https://github.com/cyx/dep

Instead of changing environment variables when $PWD changes, I went the other way: make it obvious when you're in a subshell by strapping $VIRTUAL_ENV onto $PS1. Again, simple shell scripts win[0][1].

I'm still interested in a cross-language approach, but I'm suspicious of anything which requires an explicit import step for rubygems that doesn't involve resolving the dependency tree. Unless Gemfile.lock is involved, it's going to need to copy Bundler's resolution algorithm, which seems fraught.

[0]: https://github.com/regularfry/gemsh [1]: https://github.com/regularfry/rv

>but I'm suspicious of anything which requires an explicit import step for rubygems that doesn't involve resolving the dependency tree.

The current 'guix import gem' utility doesn't do dependency resolution at all. It just generates some Scheme code for you to start with and you fill in the gaps. It's important to understand that Guix packages specify the exact versions of its dependencies. That includes not only the version number, but its explicit dependencies, the Ruby used during the build, the GCC used for native extensions, the full build script, etc. All of these various inputs to the build function are reflected in the SHA256 hash that uniquely identifies that package.

The loose versioning that Bundler uses is indeed very flawed and prone to strange errors. Guix simply doesn't have this issue.

I do something similar to this. My right prompt shows a small fork symbol when I'm in a sub-shell (based on $SHLVL). However, this does not help me much when I'm several levels deep, which is why eventually I started hooking in on changes to $PWD.

Nothing about rbenv in the post. Any reason why not? Some of the things you pointed out about rvm would be handled by using rbenv instead.

I should have made a note that I've been using rbenv as an escape from rvm and it has roughly the same set of problems, but it is a bit less enraging.

As someone who tried RVM for a while and until it's problems became obvious, and lived (very successfully) with rbenv as an escape for a long time afterwords, I have been liking chruby:

https://github.com/postmodern/chruby

It doesn't have most of the problems of RVM/rbenv, and mainly provides an easy way to switch between rubies while leaving everything else as full-path references instead of the "magic" tht RVM tries to provide.

That said, I really like the local $GEM_HOME method, and guix sounds like an interesting solution.

Thank you. I think the benefits of Guix start to become even more evident when you work in a multi-language environment with god knows how many package managers. We don't yet have all the packages that people need, but that will happen in time if other people see value in Guix. We will see.

no 'ruby version manager', eg 'chruby, rvm, or rbenv' make any sense in the nix/guix world. the article would have done a disservice to itself to go into detail on each one, when the goal was really to show how to manage ruby projects in guix.

I'm starting to think that I should go back to GoboLinux from Ubuntu.

I imagine everything said in the article also applies to the Nix package manager[0]. One of its nice features is the Nix expression language (aka Nix) [1]. It has an elegant mixture of purity and pragmatism. I imagine guile scheme would have a lot more integration opportunities outside of just system/build/configuration management. It'd be interesting to see Guix expressions that actually mix other scheme libraries or concepts, rather than just taking Nix expressions and removing the syntax.

[0]: http://nixos.org/nix/ [1]: http://nixos.org/nix/manual/#chap-writing-nix-expressions

>It'd be interesting to see Guix expressions that actually mix other scheme libraries or concepts, rather than just taking Nix expressions and removing the syntax.

We already do this. Our build scripts are written in Guile, not Bash like in Nix, and they take advantage of Guile's standard library which has things like a pattern matcher, an HTTP client and server, a POSIX interface, a foreign-function interface, an XML parser, etc. We use third-party libraries such as guile-json and guile-charting in some of our tools, as well.

I very much like how Guix is just a Guile Scheme library and can itself be integrated with other Scheme code. There is, for example, guix-web, a web interface to the package manager; there also is an Emacs interface using geiser.

Using Guile has turned out to be a great feature in itself and it is still fueling the development of new features that otherwise would be awkward to implement, such as the new "guix graph" command to visualise dependencies.

> I'm not sure how you feel, dear reader, but my Ruby environments feel like one giant, brittle hack

Of course things are not perfect (and guix may have a few cool ideas), but I think we should keep in mind that both rvm and bundler have been huge improvements.

The hacks people needed to come up with before rvm and bundler existed were an order of a magnitude more brittle.

Some of these issues can be resolved using Ruby Install https://github.com/postmodern/ruby-install

i think guix/nix solution (can) goes beyond ruby platform and you can define dependencies on non ruby components: postresql, mysql, mongodb, ...

I really hope no one is ever doing `rails >= 4.0` in their Gemfile.

Yeah, while I think the approach outlined in the post makes sense and could be really great, I really haven't had any of the problems outlined since the bad old `config.gem` days. In my experience, everyone defaults `~> X.Y.Z` dependencies, with more lenient patterns justified by actual testing, and I can't remember the last time I saw a blanket `>= X` requirement. This has meant that all, or very nearly so, of the dependency problems I find myself handling are due to actual incompatibility, which I'm glad were caught.

NB, this is a post by David Thomas, not Dave Thomas, the longtime and well known Rubyist.

Thompson, actually. I am not the Wendy's guy. :)

Oops, sorry :)

Reply on news.ycombinator.com