Hacker News Reader

Comment by gnarula | Hacker News Reader

In case anyone's curious, in absence of a shebang (#!), Linux [1] returns an ENOEXEC to the execve syscall, after which the invoking program (the shell) handles the failure. Usually shells default to running the file as a shell script with itself as argv0.

[1] https://github.com/torvalds/linux/blob/3e732ebf7316ac83e8562...

View on HN

This can lead to an interesting situation where a program will work if launched by a shell (or by another program that uses `execlp`), but will fail if it is launched with a different variant of `exec`. For example:

    $ touch empty
    $ chmod a+x empty

    $ ./empty  # Works

    $ valgrind -q ./empty  # Works

    $ timeout 10s ./empty  # Works

    $ /usr/bin/time ./empty  # Works

    $ perl -e 'exec("./empty") or die'  # Works

    $ python -c 'import subprocess; subprocess.check_call("./empty", shell=True)'  # Works

    $ python -c 'import subprocess; subprocess.check_call("./empty")'  # Fails
    ...
    OSError: [Errno 8] Exec format error

    $ ruby -e 'exec "./empty"'   # Fails
    -e:1:in `exec': Exec format error - ./empty (Errno::ENOEXEC)
            from -e:1

    $ strace ./empty  # Fails
    execve("./empty", ["./empty"], 0x7fff6639f3a0 /* 84 vars */) = -1 ENOEXEC (Exec format error)
    strace: exec: Exec format error
    +++ exited with 1 +++

It can be quite fun to track down why a program executes successfully and then later fails to execute, with no changes made to the program in between.

I knew this about the programming languages which wrap execution in a shell, but I never knew that execlp/execvp/execvpe handled ENOEXEC internally and wrapped the command in a shell too. TIL!

A fun consequence of this is that running a python script that doesn't have the shebang will mysteriously hang with a crosshair cursor until you click the mouse. When the shell encounters the first "import foo" line it tries to run ImageMagick's "import" utility which tries to take a screenshot of the selected window.

Was this behavior inherited from Unixes past? It seems like it'd be better if shells just returned the error back to the user.

Shells return exit codes, not typed error values. What kind of output would you expect from the shell, exactly?

Status 129 or something else outside the usual range, plus a message that the file was not a valid ELF and also has no shebang, therefore cannot be executed.

I always thought that 128+ are for (shell-spawned) processes and 0-127 are for shell. But, apparently POSIX requires this behavior for empty files - we need to amend the standard, first (or fail to comply, muahahaha).

I'd expect the same as any other failure from execv(2). Something like,

  zsh: Exec format error: empty-file
  (last command returned 127.)

(127 is the exit status for other failures, so I've used it here. The second line is intended to be part of my PS1, the first zsh's normal reporting. The string used here is what I get from perror() for that code.)

Okay, yeah. That could be useful. Seems POSIX forces the behavior described in the blog post. Time to change the standard!

Yeah, it predates the addition of shebang support in BSD.

From POSIX [1][2]:

> If the execl() function fails due to an error equivalent to the [ENOEXEC] error defined in the System Interfaces volume of POSIX.1-2017, the shell shall execute a command equivalent to having a shell invoked with the pathname resulting from the search as its first operand, with any remaining arguments passed to the new shell, except that the value of "$0" in the new shell may be set to the command name. If the executable file is not a text file, the shell may bypass this command execution. In this case, it shall write an error message, and shall return an exit status of 126.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V... [2] https://unix.stackexchange.com/a/373229

Thanks!

Does POSIX define what a "text file" is?

If it does, I never found it.

> A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. Although POSIX.1-2017 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files. The standard utilities that have such restrictions always specify "text files" in their STDIN or INPUT FILES sections.

https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...

Does an empty file contain characters? That first sentence seems like it contradicts itself.

I think the implication is that empty files are considered text files.

Yes. It contains zero characters.

An empty file actually contains zero of everything in the universe.

The idea behind the definition is that it defines set of files that will not trigger various bugs in traditional unix text tools implementations (ie. various variants of not checking the fgets() return value).

What is also interesting is that while strace seems to report much less overhead in "./empty" compared to /bin/true

   $ strace -c ./empty 
   strace: exec: Erreur de format pour exec()
   % time     seconds  usecs/call     calls    errors syscall
   ------ ----------- ----------- --------- --------- ----------------
     0,00    0,000000           0         1         1 execve
   ------ ----------- ----------- --------- --------- ----------------
   100,00    0,000000           0         1         1 total


   $ strace -c /bin/true
   % time     seconds  usecs/call     calls    errors syscall
   ------ ----------- ----------- --------- --------- ----------------
    54,05    0,000060          15         4           mprotect
    24,32    0,000027          27         1           set_tid_address
    10,81    0,000012          12         1           set_robust_list
     8,11    0,000009           9         1           munmap
     2,70    0,000003           3         1           prlimit64
     0,00    0,000000           0         1           read
     0,00    0,000000           0         2           close
     0,00    0,000000           0         8           mmap
     0,00    0,000000           0         1           brk
     0,00    0,000000           0         4           pread64
     0,00    0,000000           0         1         1 access
     0,00    0,000000           0         1           execve
     0,00    0,000000           0         2         1 arch_prctl
     0,00    0,000000           0         2           openat
     0,00    0,000000           0         2           newfstatat
   ------ ----------- ----------- --------- --------- ----------------
   100,00    0,000111           3        32         2 total

the multiple execution seems to be much slower with the empty executable than with /bin/true using the multitime tool https://tratt.net/laurie/src/multitime/

   $ multitime -n 10 ./empty
   ===> multitime results
   1: ./empty
               Mean        Std.Dev.    Min         Median      Max
   real        0.006       0.000       0.005       0.006       0.006       
   user        0.001       0.001       0.000       0.002       0.002       
   sys         0.000       0.001       0.000       0.000       0.002       
   
   $ multitime -n 10 /bin/true
   ===> multitime results
   1: /bin/true
               Mean        Std.Dev.    Min         Median      Max
   real        0.002       0.000       0.001       0.002       0.003       
   user        0.001       0.000       0.001       0.001       0.001       
   sys         0.000       0.000       0.000       0.000       0.000

> What is also interesting is that while strace seems to report much less overhead in "./empty" compared to /bin/true

That's because it reports an error and does not actually execute the file.

>strace: exec: Erreur de format pour exec()

`strace` uses one of the exec()-style functions that don't automatically invoke a shell here. And if it did, it would invoke an entire shell (/bin/sh, typically), which might have more overhead.

----

You need to be very careful when measuring this, because what happens is that a shell executes the shebangless empty file.

If you try launching it from e.g. a C program using execlp(), it will have to first start that shell.

If you try launching it from e.g. bash, it might directly run that in-process or at least after a fork() or similar, skipping some of the shell setup.

So the overhead might depend on context.

Interesting!

Reply on news.ycombinator.com