Sbang lets you run scripts with long shebang lines | Hacker News Reader

Sbang lets you run scripts with long shebang lines

github.com

30 pointstgamblin32 comments

I do not think that _this_ was or will be ever an issue to myself or any system that i have to deal with and i do not see any use for it but i guess somebody else did. Anyway I believe this page is a very nice read on the background and a little history https://www.in-ulm.de/~mascheck/various/shebang/

This reminds me of one of the _very few_ bits of freebsd userspace that are better than gnu and was only adopted by gnu much later (8.30, the version first in debian buster, I think): `env -S`.

`env -S` introduces an option parser/splitter for #!-lines which can do just about whatever you need (however, it will not work around over-long lines like sbang proposes to do)

So, taking the example from the GNU man page:

  #!/usr/bin/env -S perl -w -T

Why is the `/usr/bin/env -S` even in there? Why not rather

  #!perl -w -T

I know that not all environments support non-absolute paths to interpreters like this, but it feels to me that adding -S to env is a worse hack than allowing non-absolute paths to interpreters. I’m not sure which platforms support resolving the interpreter by PATH (actually, is supporting it a shell thing?), but it may even be more portable than -S at present, GNU coreutils env only having had it for two years. And even /usr/bin/env itself isn’t fully portable, as there are (old) platforms that have env at /bin/env and not /usr/bin/env.

It's there because on some systems all the args are passed as one. -S splits them out. You can try this on Linux and env even warns you about it:

    # cat test.py
    #!/usr/bin/python3
    import sys
    print(sys.argv)

    $ cat test
    #!/usr/bin/env test.py foo bar baz

    $ PATH=. ./test   
    /usr/bin/env: 'test.py foo bar baz': No such file or directory
    /usr/bin/env: use -[v]S to pass options in shebang lines

with -S, env does the right thing, and you get this:

    $ cat test 
    #!/usr/bin/env -S test.py foo bar baz

    $ PATH=. ./test
    ['./test.py', 'foo', 'bar', 'baz', './test']

You completely missed the point of what I was writing. I’m not talking about why -S, but why /usr/bin/env -S.

Yeah I guess I read that too quickly -- sorry.

This:

    #!python3

Seems to work (i.e., search $PATH) on macos but AFAIK nowhere else. It fails on both ubuntu20 and centos8. Per this: https://www.in-ulm.de/~mascheck/various/shebang/, relative paths are supposed to be interpreted as relative to ., so I'm guessing that is why everyone uses /usr/bin/env.

Given that shebang is implemented in the kernel (and not in user-space like coreutils) I think env is more likely to change than the shebang mechanism, though who knows -- there have been recent changes to shebang so perhaps further work on this will help: https://lwn.net/Articles/779997/.

The shell you’re using matters, too; I find that PATH search happens when starting things from zsh, but not bash or sh. That might be why you observe it working on macOS, I dunno—they switched to zsh a while back, didn’t they? I’m not sure how much is in the kernel and how much is elsewhere, but I do observe that the shell is at least partly responsible for the error message like `bash: /path/to/script-with-shebang: python3: bad interpreter: No such file or directory`.

The kernel doesn't handle path variable. Userspace does.

I've never had this problem, what is the use case? I've only ever used /usr/bin/env or /bin/bash, and can't think of a reason to do anything fancier. You would be hardcoding the full (presumably non standard) path to the interpreter right? Why not just add it to PATH and use env?

See shortenPerlShebang in https://nixos.org/nixpkgs/manual/#sec-language-perl

Darwin-only by my read.

In Spack https://github.com/spack/spack), we encounter this when users install packages in deep paths within their home directories, or in deep paths within shared project directories (e.g., in NFS or Lustre on HPC machines). So we patch installed scripts that have long shebangs.

We want installed packages to work exactly as built, with the right versions of dependencies, and we don't want to rely on the user getting their environment right to do that. Spack users may install several versions of python or other interpreters. This ensures that scripts work without a special environment.

There is another use case that is less emphasized here. sbang also lets you pass arbitrarily many arguments on the shebang line. If you do this:

    #!/usr/bin/perl arg1 arg2 arg3

At least on Linux, you'll get one argument: "arg1 arg2 arg3". I think perl gets around this by parsing the shebang line itself, but sbang is more general. You can use it with /usr/bin/env to pass more arguments than would otherwise be allowed.

See https://www.in-ulm.de/~mascheck/various/shebang/ for an extremely comprehensive list of limitations on shebangs.

Thanks for sharing this. Wish I had this years ago at an HPC gig at FNAL (similar use case, with users having deep path home directories on Lustre and pNFS).

> There is another use case that is less emphasized here. sbang also lets you pass arbitrarily many arguments on the shebang line. If you do this:

Most often when calling programs with a shell script it is recommended to make use of env to find it.

Like...

    #!/usr/bin/env python3

As in the limitations link above, env is pretty much everywhere, and when it isn't, you're probably going to struggle anyway to make anything portable to those systems.

env actually has an in-built flag for ensuring you're passing arguments.

    #!/usr/bin/env -S python3 -m http.server 8010

True. The two disadvantages are:

- you still get the python3 in PATH, not a specific one (sbang aims to use a specific interpreter)

- The -S argument to env is not POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/e...

> you still get the python3 in PATH, not a specific one (sbang aims to use a specific interpreter)

env lets you solve that one too. You can control PATH. From the FreeBSD manpages:

    #!/usr/bin/env -S -P/usr/local/bin:/usr/bin perl

And if you lack the -P flag... env controls the environment variables... So, you could extend (or replace) it like so:

        #!/usr/bin/env PATH=/home/my/loc:${PATH} perl

> The -S argument to env is not POSIX

You're right. POSIX only guarantees the -i flag, and the ability to set variables. It hasn't been modified in the standard since 1991, despite other updates to the standard.

OpenBSD and Busybox only supports the crippled POSIX version. It isn't completely portable.

Versions with -S and are available on Linux, FreeBSD (also has -P), macOS (also has -P).

Yeah it would be nice if POSIX would move a little faster, but I guess the slowness is the point.

This is probably obvious but bears repeating since we're on this tangent -- even -S and -P do not prevent the OS from truncating the shebang if it's too long. So env is great for passing multiple args, but it still doesn't let you use really long shebangs (the original point of the thread :)

Slightly off-topic, but are you aware that "spack" is a common derogatory slur in British English?

Why not just output a wrapper script? That seems to achieve the same thing while adhering better to the principal of least surprise

I remember running into this a few years ago with a python virtual environment created in a deeply nested directory, where (IIRC) scripts installed by pip would wind up with the virtual environment’s python path as the shebang. These failed on some of but not all of our boxes. I forget how we resolved it, but I do remember being shocked to discover that there was a length limit.

Argument cause must be pretty rare, but I can imagine it happening with long paths to a virtual environment's interpreter nested deep somewhere in a home directory.

Why not just do this:

    #!/bin/bash
    /some/long/path with lots of args

You probably want an exec in there (but I can't see a problem with this approach)

I'm not sure how this solves the problem. The goal is to execute an interpreter in a long path via the shebang line, but this is a regular script that executes a program in a long path. You can't use this to make a file invoke /some/long/path when executed. Or am I missing something?

Yeah, I see now what it's doing (and how it's implemented), so the use case makes more sense.

The problem with shebangs that I really want to solve is that I often need to edit files on a Windows machine, and then try to run the resulting CR/LF scripts on a Linux machine using a shared filesystem. (Docker, WSL, Vagrant etc...).

I'd like to invoke them with just (eg.) ./dostuff.py - Python, Ruby etc... have absolutely no problem running files containing Win-style line-endings. The only issue is that /usr/bin/env complains that it can't find (eg.) "python3\n".

Yes, I know I could convert these files with dos2unix, and I also know that I can just invoke the interpreter explicitly - but I'm lazy, and this seems like such a trivial thing to solve that I can't believe it's not been done already.

I've taken to actually recompiling /usr/bin/env to strip trailing whitespace from the executable name - but there must be a better solution.

There are editors that allow preserving the line endings (be they LF, CRLF or CR) from the source file. That could be a solution since you already seem to have control over your environments.

Yeah, this is absolutely true, but we don't really like to impose a rule regarding which editor(s) people use. Most of our workflow is Windows-based.

I wonder if this could be part of binfmt_misc if Linux let you be the handler for shebangs in general?

I think the easier method would just be defining an alternative character sequence to shebang lines which binfmt_ misc is happy to do for you. You already have to change the shebang line to support long lines, what’s switching the symbol to ‘#%’ or something?

Wouldn’t it be better to standardize format and baseline argument spec, and let the evaluating environment figure out the rest? Formalizing the entry path is something that is now becoming two lines of indirection just to lie to whatever is actually interpreting it. It’s another line of implicit code that’s up to interpretation and can easily diverge.

> Wouldn’t it be better to standardize format and baseline argument spec

Better, yes. Easy, no. Impossible is a sort timeframe, probably (given how many systems are out there not confirming to this as yet undecided standard).

Maybe that will actually happen long-term, with this being essentially a polyfil.

Reply on news.ycombinator.com