Hacker News Reader

Comment by kstrauser | Hacker News Reader

From https://www.gnu.org/software/stow/ :

"GNU Stow is a symlink farm manager which takes distinct packages of software and/or data located in separate directories on the filesystem, and makes them appear to be installed in the same place."

The idea is that instead of installing package foopkg directly into /usr/local, you could install it to /opt/foopkg-v1.2.3. Then you can run stow to make a bunch of symlinks like /usr/local/bin/foo -> /opt/foopkg-v1.2.3/bin/foo. Upgrade it to a new version, re-run stow, and now all the symlinks point to /opt/foopkg-v4.5.6/bin/foo and so on. It's pretty nifty.

However, I used it more for managing dotfiles in my home directory than anything else, making links like ~/.vimrc -> ~/src/my-config-repo/.vimrc . I much prefer using chezmoi for that now.

View on HN

I've found that stock git works great for managing dotfiles, without any extra tools needed. Just a few lines of gitconfig and a shell alias is enough. It's all explained here: https://www.atlassian.com/git/tutorials/dotfiles

Perhaps other people have more complex use cases than me.

There's nothing wrong with that setup. It falls over when you start pushing it across multiple machines with substantial differences. Then Chezmoi's templating is so handy. For example, 99% of my config is the same between my desktop Mac and my various Linux servers, but I use different ssh_config settings on the 2 OSes. Chezmoi makes that very easy. Stock git doesn't. I could script something up to do handle that for me automatically, and before I knew about Chezmoi, that's exactly what I did. Now I'd prefer to let someone else write and maintain all that for me so I can move on to working on other things.

That's what branches are for. I have two personal machines and two work machines, all of which have diverging configs, which I push to two remotes (one work specific), and I merge changes between them.

This is how Git was designed to be used.

Until you want to introduce a change that affects both machines. You need to start rewriting history or cherry-pick the changes on both branches. The further the history diverges, the harder this becomes.

Using branches for this does not scale.

This is where a VCS like Pijul or Darcs would shine, since patches commute across "branches" without a new hash.

How does that work if there's a conflict? Whether or not there's a hash involved, you still have to manually apply the patch to each branch, would you not? If there's no conflict, merging each branch up in git is not hard at all, but it's still a tedious extra 'n - 1' operations that you don't need in chezmoi.

For example, say I have a line "export FOO=bar" in my .bashrc on one machine and "export FOO=baz" on another. If I then indent the line on the bar branch and try to merge to the other one, something has to tell the baz branch that the right line is both differences: " export FOO=baz". Except the conflict may not be so obvious to resolve as that! And whatever you do, you'll either have a "trellis" of 'n' branches if you merge, or 'n' parallel linear branches if you cherry pick everything. Both of those history layouts quickly become very hard (to me at least) to make sure they all contain everything they should and nothing they shouldn't.

Whereas with chezmoi, the bashrc file is a template that is the same on all machines that simply says "export FOO={{ fooval }}" and chezmoi does the templating. So you can just indent the line and fast-forward/apply on other machines and that's it.

It's not intended as a dotfile manager replacement, but a Git replacement.

As long as the patches don't conflict its fine and dandy, if there's a collision you record a resolution that fixes the conflict

Also, the conflict resolution is just another patch (Pijul patches aren't just regular diffs, they have a lot more information), so should you decide to merge it back upstream after all, you can also cherry-pick the conflict resolution along with the conflicting patch, and also without changing the hash.

One of the motivations behind Pijul was to manage custom versions of Nixpkgs while still benefiting from upstream commits. One issue that's hard with Git is that when you also want to contribute multiple changes back, you have:

1. A branch pointing to the latest nixpkgs head.

2. A branch with commit A (let's say commit A introduces a new package to nixpkgs).

3. A branch with commit B (changing some config file).

4. A branch currently at in use for your own machines, with branches 2 and 3 rebased on top of branch 1.

Every time you do anything, you'll have to remember the flow for getting the commits fetched/rebased. Which is fine if you have a DevOps team doing exactly that, but isn't too cool if you are anything other than a large company.

In Pijul, you would have a single channel (branch sort-of equivalent) and two patches (A and B) instead, which you can push independently from each other at any time if you want to contribute them back.

Darcs does the same but wouldn't scale to Nixpkgs-sized repos.

I have no idea what you're talking about. You just merge the change.

I was bitten by merge conflicts many times with such workflow. Not anymore

With `Match exec ...` you can define arbitrary machine-specific sections.

I had more problems with differences between versions than with differences between machines. tmux is the sort of program where the available config directives change every major release and there is no full backwards compatibility.

I evaluated Stow and tried Chezmoi for a while but settled on YADM. It’s the bare git repo idea with a little more sugar sprinkled on top. Perfect for my needs.

I switched from Yadm to Chezmoi.

The main reason was because there invariably is always something slightly sensitive you don't want in a dotfile and the rest of the file is okay. Yadm uses third party tools to do jinja templating. The first one envtpl stopped being maintained, and the second one j2cli (both jinja2 templaters) aren't very well maintained either.

With chezmoi I just use the golang text/template templater. I know it will always be maintained. The integrated password manager functionality for chezmoi also works awesome too.

I did initially use stow, but symlinks is just bad. You end up with all sorts of problems with that I can't even remember. My whole dotfiles is 7MB, so if a copy is made from a "source tree" to my home dir that's okay.

Chezmoi also encouraged me to do things more deterministically based on hosts and reduce the number of "scripts" that I run significantly, which led to less bugs. I use the same set of dotfiles across a number of my systems.

I use yadm with the default templating system which is based on awk...

https://yadm.io/docs/templates

i like yadm because it has simply no dependencies and can be installed literally anywhere on any archs (which is important to me)....

the yadm/awk templating system is good enough for me, it let you do some if host then output this or output that kind of things..... i never had a need for more.

https://yadm.io/

I think I heard about it before, but haven’t tried it.

https://www.chezmoi.io/

I currently use Borg in Vorta to manage my dotfiles. I only have about 5, and half of them have secrets.

I could do it all with shell... but every few likes of custom scripting and config I can ditch is one less thing to worry about, and I don't really need VCS for just a few kb that rarely change.

I've been using a lot more VSCode extensions lately though, so perhaps I'll want to do something for that.

yadm https://yadm.io/ uses git for file management, but also provides some convenience on top of that

Can recommend this setup as well, it's great.

>It's pretty nifty.

It's how Amazon's stuff used to work (though not using Stow) back several years ago. No idea if they've migrated from that approach to containers, or similar, yet.

Every application you deployed would have the necessary components deployed (or re-used if you had something else that already used it), and then build the application space from symlinks to those parts. Worked really well.

Amazon has an internal Nix/Guix? Probably not much public info on this...

It's not the same.

It's been a while, so I'm sure to get things wrong, but you basically had different package groups you could set up. So a service would have its group that it could update, test, and deploy with at once.

But like if I produced a package, and another team depended on it, there was no guarantee that the group I ran in CI had versions in common with the group that they deployed.

I also remember some weirdness just within one package, like maybe your PR build was based on your local group setup and not anything "official".

The coolest thing about it is that you could make a PR against multiple repos at the same time, even if one depended on the other. Like you could add a function to a library in one repo and call it from another repo in one PR.

Alternatively, Amazon is using stow, a common GNU utility whose info page refers to a version of Perl released in 1992, or something similar to it, instead of Nix and Guix which didn't exist when Amazon started.

It used neither, but it's own implementation of the concept ("symlink all the things!", which is much older than both, and has lots of other implementations too, aside from Nix and Stow.

It's how gobolinux works too, for example.

Would Amazon rely on stow when it is in this peril? Unless they really do rely on it and maintain an internal fork, which would make this situation even worse. Or they use it anyway...

Which is also the software this submission is about

Yes. From what I was told, it was already of reasonable vintage when I joined in 2013.

Still works on link farms, yep. And works pretty well!

Ah, symlink farms, how I love to hate thee! They are alive and well.

Why do you hate them?

(FWIW this is a sincere question; given the number of these things I touch, I would very much like to know if there are problems I need to know about and/or better alternatives)

What is the use case for this? Is the idea that it can automatically turn a package into a portable package, effectively? Or is it so you can install multiple versions of the same software without conflict between them?

You install libfoo 4.7 to /usr/local/stow/libfoo-4.7 (such that you have /usr/local/stow/libfoo-4.7/bin/foocfg and /usr/local/stow/libfoo-4.7/lib/libfoo-4.7.0.so and /usr/local/libfoo-4.7/share/man/man1/foocfg.1.gz), and then libfoo 4.8 to /usr/local/stow/libfoo-4.8. Then from /usr/local/stow you run `stow libfoo-4.7` and all the contents mentioned above are symlinked into /usr/local appropriately. Then if you want to switch libraries you unstow that one and stow version 4.8.

It's highly configurable, so you can do a lot more with it than that, but that was the idea behind it 25 years ago. There were whole distros based on that, though it fell out of favor when containers became a bigger thing in the late oughts.

It lets you have informal package management of self-compiled binaries in parallel with your distro's package manager. With Stow you can install updated libs and applications into /usr/local and don't have to be concerned with conflicts. At worst you may need to set LD_PRELOAD to bypass system libs. Very useful with Debian stable when you need a new feature in something and don't want to wrestle with backports.

I've never had a problem installing self-compiled stuff into /usr/local without conflicts. I'd have to have the problem of installing multiple versions (upgrades) and needing to be able to roll back easily, along with any config files and such.

The problem is when you want to uninstall things. With Stow you don't have to track down the installer droppings scattered everywhere. The nice part of Stow is that it builds minimal symlinks and converts them to a deeper hierarchy once a second package wants to use a common directory.

In the distant past, /usr/local/stow was a NFS mount. Each machine could maintain different symlink trees.

And this sort of thing is still common in large shared research computing clusters, where stuff gets installed in arbitrary locations and old libraries with obscure build dependencies are the norm.

For this though, "modules" is still around [0]

[0] https://modules.sourceforge.net/

Curious: How do you uninstall stuff you manually installed in /usr/local?

I’ve used this a lot on Debian systems so that I could just use apt to remove the manually-compiled version: https://en.m.wikipedia.org/wiki/CheckInstall

These days I mostly use Nix which basically elominates this problem.

I don’t know how I never heard about CheckInstall! This is great.

Can Nix do what CheckInstall does, or do you need to manually build a Nix package for the program version that you want to install?

I mostly use nix with direnv so I never have to install anything globally and, between overrides and a minimal understanding of how to write a custom package, it’s surprisingly easy to get the tools you need for each project.

Anything I install in /usr/local is stuff is going to be something I really need, and for which there isn't a distro package. It stays for the life of the system.

If I wanted to install something into /usr/local that would be suspected of needing removal later, I'd build it with /usr/local as a prefix, but install it in a temporary directory, then make a tarball package out of that to keep track of the file list. That could be used to remove it. I could trivially generate an uninstall script by using find in the package directory to get a list of relative paths; converting each to a rm command. The uninstall script would be put into /usr/local and run from there.

> If I wanted to install something into /usr/local that would be suspected of needing removal later, I'd build it with /usr/local as a prefix, but install it in a temporary directory, then make a tarball package out of that to keep track of the file list. That could be used to remove it. I could trivially generate an uninstall script by using find in the package directory to get a list of relative paths; converting each to a rm command. The uninstall script would be put into /usr/local and run from there.

Compared to using stow, this is two orders of magnitude more complicated :-) With stow, you simply install it anywhere, and stow will make the symlinks into /usr for you. When you want to uninstall it, stow will remove all the symlinks. This way, I would install each package into its own directory. When I want to remove it, I use stow to delete all the symlinks, and then just delete the directory.

No need for "make uninstall", etc.

It's only theoretical; I've never uninstalled anything out of a /usr/local.

What Stow is doing, by the way, is better achieved with overlayfs, which didn't exist when Stow was first introduced.

With overlayfs you can specify multiple directories that are merged together. Multiple packages that are rooted at /usr/local can be mapped there with overlayfs.

And so your development environment artifacts can be linked in to the environment root just like any other package (really awesome).

Sorry I’m not super clear on how this works. Could you explain what it means to be linked in to the environment root?

Is there an overlap of functionality between GNU Stow and OSTree?

Reply on news.ycombinator.com