Using /usr/local | Hacker News Reader

local

hivelogic.com

120 pointsvectorpush80 comments

One of the things I love about the Debian ecosystem is that there's a tool that's even better than what this post recommends. Debian update-alternatives, available on any Debian-derived distro.

update-alternatives allows you to compile or unzip software to somewhere obscure and safe, like /opt, then soft-link it to /usr/local with update-alternatives --install. The benefits:

1. You can slave all dependencies to the main binary, so that they can all be removed with update-alternatives --remove.

2. You can install multiple versions simultaneously (including the system version), flag one as active, and toggle between them easily with update-alternatives --config.

3. #2 lets you upgrade to a new version (say of Java, Scala, Haskell, whatever) without uninstalling or overwriting the current version, and if the new version breaks your app you can toggle back to the known working version with a quick update-alternatives --config.

4. Keep all the files together in their own directory, like /opt/java/jdk/1.7.0_07, instead of strewn all across /usr/local, /usr/bin, /usr/lib, /usr/share, etc. Soft link them instead into their appropriate locations.

I created update-alternatives install scripts for Haskell Platform [1], Java [2], and Scala [3] for anyone interested in seeing more how it works.

[1]: https://github.com/byrongibson/scripts/tree/master/install/h...

[2]: https://github.com/byrongibson/scripts/tree/master/install/j...

[3]: https://github.com/byrongibson/scripts/tree/master/install/s...

I remember seeing a different tool which takes care of all that for you so you don't have to write scripts, but I forget its name now. It's definitely a good idea (slotting software installs) but it needs more automation.

I only use the scripts to slave all the dependent binaries to the main binary. If you only have one binary, it's just a single line. (I also script literally everything, even single-liners).

I don't know if there's any way to do something like that without update-alternatives or your tool having some built-in awareness of the package's dependencies, but that's not really possible for a tool designed to be able to handle literally any arbitrary, user-compiled/installed package. Otherwise you get the system package manager again.

The tool I remember would basically do the equivalent of:

  for file in `find /opt/package_name/version/` ; do
      NEW=`echo $file | sed -e "s/^\/opt\/package_name\/version\//"`
      ln -s $file /usr/local/$NEW
  done

Where the base files are installed in /opt/package_name/version/ and all of them are symlinked into /usr/local/ by default with the option of changing individual file links. This way you change the entire 'installed' version of a package without needing application-specific path modifiers. But you can still build applications against specific versioned directories (if you have an app which requires BerkeleyDB 4.1 which conflicts with 4.0, for example)

(You can automate that for a pre-existing package if it's got relocatable code, but hardly as reliable as building it from scratch for the versioned directory and changing symlinks in /usr/local)

You may be referring to paco[1], which has always been kind of a hack. It also appears to not have been updated in a very long time.

1: http://paco.sourceforge.net/

Possibly... I don't remember the tool I saw using an LD_PRELOAD, and I think it operated on post-make-install files (like a DESTDIR staging directory).

Anyway, the mini script I provided below is like half the functionality you would need after the source is compiled to a PREFIX of /opt/package/version/. Perhaps i'll write a tool when I get bored :-) The tool and method i'm borrowing from a proprietary tool I saw once, but obviously other people have implemented it too.

Paco has a great idea though, and that's a meta-format and wrapper for different system packages. If you have to support five different production operating systems and the same software on all of them, switching between package managers is a pain in the ass. An extra tool to cross-reference paco to the OS's packages could make it really handy for resolving software updates in a unified way.

> Specify the /usr/local Prefix in Scripts

> When writing scripts, specify the full path to the executable you’ve installed in the first line of the script.

No. Don't do that. Bad. Even for maintenance and admin scripts, that makes them severely less portable and harder to keep in sync across machines for various reasons. Your path is setup correctly, just use that: there are very few reasons for your shebang lines to be anything other than:

    #!/usr/bin/env $executable "$options"

This will make the shebang work using the normal rules of path-resolution, and means path-override tools and methods (e.g. python virtual environments, or `PATH=$somedirectory ./script`) will work correctly.

edit: I am told not all versions of shebang/`env` support passing options to the $executable, and most notably it doesn't work on Linux. So don't do that, and i won't in the future. Still, $executable is usually sufficient

While I agree about the use of /usr/bin/env, you should be aware that some systems will perform only one word split on the shebang line. That means that if you use env, you can't pass any options to the $executable.

You can try it if you want, save this into an executable file (test.awk):

    #!/usr/bin/env awk -f
    { print $2 }

Then run:

    echo "Hello world again" | ./test.awk

I get the following:

    /usr/bin/env: awk -f: No such file or directory

> I get the following:

Damn. I get "world" (OSX, zsh, BSD env)

So I supposed you can't pass options to `env` either (e.g. `-i`, or ENVVAR overrides?)

This is a kernel issue, it has nothing to do with userspace tools. For example, the Linux code responsible for parsing shebangs is here:

http://git.kernel.org/?p=linux/kernel/git/stable/linux-stabl...

Interesting, thanks for the info. I wouldn't have thought it so low in the architecture.

Linux doesn't support options, unfortunately. The executable path /usr/bin/env is parsed, then the remainder of the line is used as a single argument, spaces and all:

  $ cat x.py
  #!/usr/bin/env python -i

  $ ./x.py 
  /usr/bin/env: python -i: No such file or directory

+1.

At a previous job, I spent hours rebuilding a ruby gem where someone had carelessly hard-coded "#!/usr/local/bin/ruby" into all of their executables.

Even if it looks like it works, don't do it! Stick to "#!/usr/bin/env ruby" for maximum compatibility, please.

Coming from a c and python perspective, I'm amazed that you even can specify which ruby to run it with. Either its a compiled binary, or you call the interpreter with the code object as an argument. What is the design impetus for mixing scripting and executable styles this way?

Shebangs (#!) have been around for quite a long time. It's actually really handy to be able to write a script that you can execute as a command. See https://en.wikipedia.org/wiki/Shebang_(Unix)

> Coming from a c and python perspective, I'm amazed that you even can specify which ruby to run it with.

Erm... you can do the exact same thing in Python, you know...

Was this a publicly available gem? Did you pass your feedback upstream?

It was an obsolete gem that our obsolete code still depended on. There's reasons I don't work for that company anymore :)

It's a little more complicated than that. Using /usr/bin/env in your shebang is effectively a way to say "I don't care which executable you run this with". If the executable doesn't actually matter, that's fine - bash scripts probably don't care whether bash is /bin/bash or /usr/bin/bash, so it makes sense to just use whichever is in the path.

Once you start getting into Python/Ruby/Perl/etc scripts, the executable actually does matter and relying on env can make your script fragile with respect to path changes. Different executables have access to to different collections of installed libraries, and it's important to make sure that your scripts are run using the executable that has their dependencies installed. If you use env shebangs, there's no way to do that - you're limited to either using the first executable on the path or manually specifying the executable on the command line to bypass the shebang.

I'm not saying that you shouldn't use env shebags, because they really are fine for a lot of scripts. However, you should always be mindful of what they mean and how that could break your script in the future.

My experience is exactly the opposite wrt fragility: fully hardcoded paths assume that the hardcoded path can and will have all the required dependencies, which — at least for modern Python development — is always the wrong assumption to make. Essentially, it only ever works if the script is run on the same version of the same distribution with the same package manager and installed packages as the original developer of the script. This is the most brittle a script can get.

> it's important to make sure that your scripts are run using the executable that has their dependencies installed.

Which a hardcoded shebang path does not do, it assumes these things but does not assert them and does not "make sure" of anything.

> If you use env shebangs, there's no way to do that

There's no way to make unwarranted assumptions is about as far as it goes: using env shebangs, the caller of the utility has control over what happens and which executable gets called (even indirectly)

> you're limited to either using the first executable on the path or manually specifying the executable on the command line to bypass the shebang.

Or specify a PATH when calling the script, or run the script in overridden PATH environments (e.g. virtualenvs), or...

> I'm not saying that you shouldn't use env shebags, because they really are fine for a lot of scripts.

And I went (and would still go) much further and state that they're the right solution for the vast majority of scripts and utilities.

> However, you should always be mindful of what they mean and how that could break your script in the future.

So should you be with hardcoded shebang paths, and they are much more likely to break a script across porting than env shebangs.

Most of the Linux distros I've used have the generic executable linked to a specific version: ls -l /usr/bin/python -> python2.6

If you need a specific version because of libraries or whatever, just use: #!/usr/bin/env python2.7

Nice tip!

I've read better articles on Unix. Just a few thoughts:

1) Software packages should probably install into /usr/local by default, not /usr as the author suggests. Distributions can always pass an option to install into /usr in their build scripts. In my experience as a distro packager, plenty of package-build/install systems install to /usr/local by default (autotools, python's disttools IIRC, etc).

2) FreeBSD (and likely other *BSD OS) ports install into /usr/local. So, if you're using the ports system for package management, you might still be screwed. (Of course, if you, as an admin, don't notice that you're installing over system packages, your system is going to end up hosed some day anyway.)

At the end of the day, go ahead and use /usr/local. But be aware what your system does with it, and that other 3rd party packages may randomly scribble all over it as well.

Regarding point 2, the ports system works a little bit differently to typical package managers, and easily accomodates running customised versions of the software you need.

Another difference between FreeBSD and (most) Linux distros is the concept of the "base system". Things in /bin and /usr/bin are part of the base, any additional packages you install end up in /usr/local. This has the nice effect of separating things you really, really need to work for the system to be functional (i.e. boot and log in) and software that is "nice to have" but not strictly necessary (your webservers and Rubies and what have you). I also find that /usr/local/etc is generally very small and tidy, which makes administration and backup a doddle.

I think the author of the post is a little bit muddled on what /usr/local is for, though - the analogy to installing applications to ~/Applications on OS X is completely flawed, as in OS X that installs things only for the current user, whereas /usr/local is available to all.

True, the linux equivalent would be ~/.local . What I usually do for software I need only for my user is to install it in ~/.local/<package name> and then symlink what I need into ~/.local/bin,lib,etc.. with bin being on my PATH. If the binary needs to see its own libs, I write a small script that sets LD_LIBRARY_PATH prior to execution and use that. This way removing is simply rm -r ~/.local/<package> and I don't pollute the rest of the system.

ElasticSearch does everything (including config) in /usr/local/share/elasticsearch in the vanilla (not distribution specific) install.

Reference:

Personal agony.

I see a lot of comments lamenting the downfall of HN recently and they almost always focus on poor comments. In my opinion the troublesome trend on HN has been stories like this (or yesterday's bash redirection cheat sheet) making it to the front page. This is Hacker News not Un*x For Dummies.

Meanwhile, outside your ego, some of us actually found that useful, unlike all the reposted tech crunch, naive life advice and "how to get funding like a baws" articles.

YMMV

I'm sorry, I did not mean to be egotistical. I guess I assumed that no "hacker" had typed `cd /usr/local` or `cd /etc` without wondering about origins of the names enough to search out the most cursory explanation.

For what its worth, I never read the TC links;)

The thing I think your assumption ignores is that there are infinite things that we encounter on a daily basis that have interesting explanations. Different people, even different "hackers", are going to be attracted to different topics to explore. Sharing the topics that we've explored is exactly what I come here for.

We all type all day, but we haven't all dug into how keyboards work. We look at screens all day, but we haven't all learned about monitor technology, sometimes even at the "cursory" level. Not having done it yet doesn't disqualify us from being real "hackers".

I have lurked/read hacker news for a few years before creating an account, and frankly, stories like this have always made the front page, MORE frequently in the past if anything.

If you look for an issue, you will surely find one.

Although your account is definitely not "less than a year old", I suspect the following HN guideline [1] might still apply:

> If your account is less than a year old, please don't submit comments saying that HN is turning into Reddit. (It's a common semi-noob illusion.)

[1] http://ycombinator.com/newsguidelines.html

What is with the complaints about the "downfall" of quality recently? If you don't like the article, just don't upvote it. Writing a comment for the article about how it shouldn't be here isn't serving any purpose.

There's no way to downvote an article, and as the community grows larger, simply not upvoting is not enough to stop bad articles making the frontpage.

Odd, though, that a 7 year old article can turn up as news.

This effect isn't odd at all. It is explained quite well in: https://xkcd.com/1053/

Recently? People have been singing that tune for years!

There has been an annoyingly noticable uptick in the past week or so.

Dumping everything into /usr/local doesn't even work well when you're the only user on the system: most packages don't define an "uninstall" make target, and installing multiple versions of things is often not well-defined.

See e.g. checkinstall, GNU stow, encap, and probably several more that I'm missing for examples of people trying to fix the crappiness of ad-hoc (non-distribution-based) package management.

On Arch, you can just write your own PKGBUILD.

To be fair, you can do that on any distribution, that is their point.

Don't forget to add the following line to your .bash<whatever> to make the man pages of your custom software accessible:

    export MANPATH=/usr/local/share/man:$MANPATH

This works for packages installed with homebrew and othe software that follows the standards.

Thanks! I didn't know there was such a thing as $MANPATH!

But, I don't think it's necessary. I don't have `/usr/local/share/man` in my $MANPATH (I don't have a $MANPATH at all, as a matter of fact), yet `man sl` works (and by running `sudo opensnoop` in another shell process I verified that groff indeed pulled the data from `/usr/local/share/man/man1/sl.1`)

(OS X 10.8.1)

You are correct:

  $ man man
    . . .
  man uses a sophisticated method of finding manual page files, based  on
  the   invocation   options   and   environment   variables,  the  /pri-
  vate/etc/man.conf configuration file, and some built in conventions and
  heuristics.
    . . .
  $ fgrep local/share/man /private/etc/man.conf 
  MANPATH	/usr/local/share/man
  MANPATH_MAP	/usr/local/bin		/usr/local/share/man
  MANPATH_MAP	/usr/local/sbin		/usr/local/share/man
  $

(OS X 10.8.n, for some unannounced future value of n, but AFAIK this behavior is unchanged since 10.4 Tiger)

You're right. Good to know that man checks this path by default. I'm using a custom path for homebrew, that's why I added /usr/local/homebrew/share/man to the $MANPATH.

It's a nice article and a cute history lesson... but unless you're going to be deploying that website to a production server also running OS X, you will be much better off in a variety of ways installing Lighttpd and Rails to a virtual machine and developing against that.

And you probably want to go with Nginx instead of lighttpd to stay a bit more current.

It's worth noticing the article has a 2005 date on it. I suspect there's a lot of stuff the author (Dan Benjamin of the 5by5 podcast network) would change now.

Ain't that the truth. I should probably write an update at some point.

Good catch. I was developing that way in 2005 too.

With /usr/local, though, you still have to use `sudo` - unless you're crazy and chown or chmod it appropriately. I've been a longtime advocate for $HOME/.local, which is (a) obviously writable by you, (b) on your home partition, so it's backed up with all of your data, and (c) a common enough standard (see the XDG Base Directory Spec) that it's generally easy enough to get applications to use it. Just add $HOME/.local/bin to your $PATH and you're good to go.

Given that homebrew tends to occupy /usr/local, I've grown comfortable with making my own ~/local folder with src, bin and app/server folders. I tend to use it for just about everything one might want in a /usr/local directory.

From the homebrew FAQ (https://github.com/mxcl/homebrew/wiki/FAQ)

    Can I install my own stuff to /usr/local?

    Yes, brew is designed to not get in your way so you can
    use it how you like.

    Install your own stuff, but be aware that if you
    install common libraries, like libexpat yourself, it may
    cause trouble when trying to build certain Homebrew
    formula. As a result brew doctor will warn you about
    this.

    Thus it’s probably better to install your own stuff to
    the Cellar and then brew link it. Like so:

        $ cd foo-0.1
        $ brew diy
        ./configure --prefix=/usr/local/Cellar/foo/0.1
        $ ./configure --prefix=/usr/local/Cellar/foo/0.1
        [snip]
        $ make && make install
        $ brew link foo
        Linking /usr/local/Cellar/foo/0.1... 17 symlinks created

And really, writing your own formula is easy if manual steps are easy, and if they're hard, well, writing a formula makes you do it only once, since you'll be able to reuse the formula later.

Anyway brew builds in Cellar, and symlinks in /usr/local but never steps on what you would have installed manually.

/usr/local is a crutch.

When you're a user installing software, you don't get to write to /usr/local. You have to use $HOME/usr or something and modify your path.

When you're an admin on a box, you can install software anywhere. You could totally './configure&&make&&make install' and your software lands in /usr/local. If you're lucky, there won't be any conflicts with existing earlier versions in /usr. But there's consequences.

Now you've built a lot of software and some of it depends on stuff you put in /usr/local. Now you need to upgrade. But is something going to break? What's the compatibility? How can I track what has been installed? Can I uninstall anything?

You don't realize how big of a mess you're getting into until you have to remove all of /usr/local just to get some working apps and start from scratch. Take the time to learn the package manager for your system, and package your software before installing. Hell, use checkinstall or some other tool to automate it.

I use my scripts called slacktools [https://github.com/psypete/public-bin/tree/public-bin/src/sl...] to automate packaging in Slackware. One command 'slackpack foo-1.4.5.tar.gz' builds me a Slackware package. Tiny modifications could make it build rpms, debs, etc. The benefit you get is actually upgrading old packages do you don't have conflicting versions, a sane default path, and the ability to remove or upgrade the package when necessary.

Disappointed to see no mention of GNU Stow there, very handy for /usr/local having multiple version of foo installed and being able to make one of them current, e.g. rewind if a new one breaks things. http://www.gnu.org/software/stow/

I don't think Stow existed in 2005, when I wrote the article. Thanks for the pointer.

If definitely existed before 1996 as I recall introducing it to a particular company's servers. :-)

And BSD users have been doing this forever. I know I've read about why Linux is divergent, but I forget why.

Linux isn't divergent, except for the common practice (though not quite a standard) of using /opt for non-distro package-installed software. All autotools-based packages drop themselves into a prefix of /usr/local by default, which makes the admonition to pass --prefix=/usr/local to configure a little strange.

Almost all software I've custom built over the last 13 years of using Linux has, by default, set the prefix to /use/local. I don't know if ruby is for some reason different, but installing software outside of /usr/local is not something to snark at Linux about...

No snark intended. I mean only to recognize BSD in the discussion as having made another choice. And yes, Linux custom-built software goes into /usr/local without a hitch.

But really, the question comes down to layers of abstraction. Is /bin the base system? Is /usr/bin? Is /usr/local/bin? Probably not on the last one, but Linux distros have chosen /usr/bin as where all packages install. And BSD treats packages as user programs and goes to /usr/local/bin.

I'm not saying one is right. At best, I'm advocating use of "#!/usr/bin/env foo" instead of some of the assumptions made by some shell scripters.

As I've always understood it, since /usr can be on another partition (to keep / small, or keep it from filling up with stuff from /usr, /home, /tmp, and the like), /bin should contain "essential" binaries for booting that weren't necessarily "system" tools (those would go in /sbin). Now that I think of it, I can't really think of binaries essential to booting that aren't "system" tools, but my Linux boxes typically have about a hundred CLI programs that are basic, but not essential (AFAICT) to booting in there. They're all from packages.

The way I understand it, the 's' in "/sbin" does not stand for "system", but for "superuser" - essential system tools which need superuser privileges (insmod, ifconfig, fsck, ...)

/bin should contain all the non-privileged essential tools for system to boot up and execute all its initscripts - stuff like echo, cat, grep, awk, sed, chmod and many other everyday tools. In addition, it should contain tools which will allow you to repair a broken system in minimal environment (where only / is available to you). So, stuff like ls, mount, gunzip, dmesg, ...

Apparently most modern Linux distros cannot boot anymore without /usr. Therefore people suggest to link /bin to /usr/bin etc. /usr/local still has its place, though.

http://www.freedesktop.org/wiki/Software/systemd/TheCaseForT...

I'm pretty sure Debian (my Linux of choice) can boot without /usr. And I love how on that page they bring up BS such as "well Fedora has 450MB (of crap) in /, so that's not minimal". I've got at least one Debian box packed to the gills with programming languages, scientific software and who knows what else, and it clocks in at right around 129MB for /. Maybe Fedora is doing it wrong; wouldn't surprise me, as I switched from RH to Debian over a decade ago because of dependency hell.

I'm not sure linking /bin to /usr/bin is a good idea; it may have merits, but many of the reasons they bring up seem dubious ("Solaris is doing it!", yeah like Solaris should be the gold standard; "hard coded paths!" - recompile). I'm not opposed to progress, but some change isn't progress. It's interesting to me that this initiative is being pushed by Lennart Poettering - well known for his work on PulseAudio and systemd, two other controverisal Linux subsystems. While systemd is still unproven (and quite frankly, Poettering's talk at Linux Plumbers about systemd subsuming more control scared me; it's like a hydra!), PulseAudio appears to have finally become stable and usable. I guess time will tell with both systemd and /usr-merging.

Another interesting point from Poettering's talk was that he wanted to take more after BSD, especially having the one "correct" way to do it. Me, I like having options, especially in the case of "what if the 'correct' way doesn't work?" (http://news.ycombinator.com/item?id=4409768).

Bash is well qualified for /bin.

It's the BSD approach that's divergent. /usr/local is supposed to belong to programs that have been manually installed by the administrator. BSD's awkward and frankly unhelpful split between packages and ports means they need one more level of hierarchy than linux (rather than package-managed programs and manually installed programs, you have package-managed programs, ports-managed programs and manually installed programs); unfortunately they've taken over /usr/local for the ports-managed programs, leaving you with nowhere for manually installed programs. And while ports purports to support a configuration variable for changing where ports software gets installed to, it must not get tested because a number of programs break if you try and actually use it.

I still wish this wasn't all that hard-coded. A plethora of prefixes, the whole "/usr" directory from the start (which used to be "/home" back in the days), compiling software to depend on variant paths, huge PATH variables and the hassle to define them in the proper place, application data all strewn about etc.

Yes, there's stow/homebrew like deployment with app-specific directories and linking to solve some of these problems…

But I really wish union mounts would be more common, both global and on a per-process basis. Thus you'd just have a /bin directory and just put everything in there, no matter on what partition or under which "real" directory it actually is.

> On many UNIX systems, mounting drives has become automatic, and the mount points for the drives live in a folder called /mnt.

No, no, no, no, no. This hasn't been the case since at least eight years ago, when FHS 2.3 (released in Jan 2004) clarified the purpose of `/mnt` and added the `/media` directory. `/media` is for mountpoints for removable media, and `/mnt` is for temporarily-mounted filesystems.

Ok, the article was written in 2005. I can give it some slack.

"not knowing, she accepted the default paths at compile time, and the [Ruby] binary now lives in /usr/bin" wat? Ruby's default is to install to /usr{/bin,}?? Automake and cmake at least always default to /usr/local{/bin,} as the prefix in my experience. Other than not mentioning that, yes more people should know the various prefix conventions, it would save a lot of installation hassle.

No, Ruby's default is to install to /usr/local. It was just an illustrative example used to demonstrate potential complications from not using /usr/local. The submission wasn't a guide to building Ruby.

Except doesn't Ubuntu entirely break this by putting packaged files in /usr/local? I've always use /opt for this purpose.

Then there's stow, which just works nicely.

To be honest, using the Arch Build System and quickly building my own packages beats all of this.

I don't believe Ubuntu does that. Can you provide some verifiable examples?

I use /opt/ld/{bin,sbin, ...} myself, on the grounds that it's short, yet just confusing enough to be unattractive to future filesystem standardizers. By sheer coincidence, it's now a backronym[1].

[1] http://lagrangiandesign.com/

Try "$HOME/.local". I don't think anyone's building packages that put things there.

Fairly obvious, if you've been adminning UNIX-type systems for a while, but on top of this I would add: use stow: http://www.gnu.org/software/stow/

Wow, excellent write up.

I thought /opt was the new /usr/local, no?

I remember an argument 10 years ago where a (very knowledgable) coworker swore to me that /usr/local was supposed to be world writable location for any user to install software.

He also claimed that he didn't save non-compilng code while he was editing.

Reply on news.ycombinator.com