Anatomy of a shell

I don't see sh as a losing battle. I actually think it's quite good for the niche it aims to serve - a niche that very much exists, sh is a square peg and should only go in square holes. I think most folks have had their impression of POSIX shell tainted by bad shells like bash and - sorry - oil encouraging complex, non-portable shell scripts. I intend to write more blog posts in the future helping people gain a closer understanding of POSIX shell and how/where it's useful and how to use it.

On one hand I applaud, but on the other strict POSIX shell seems overly limited.

My personal grudges:

- no local variables (even dash implements them)

- no process substitution [0] so one has to use temp files, temp named pipes or possibly joggle fd numbers. I once did an external implementation of process substitution, so it's doable with POSIX shell, but then pointless.

- set -o pipefail

I understand that you know about and get over those limitations. But isn't it too cumbersome when Linux world settled on bash and BSD on ksh? Especially since ksh is a bit like a subset of bash. At least the things I mentioned are common for both.

It's sad that shell makes it so hard to make more complicated pipelines than | the | old | classic. Fds, processes and pipelines should all be first class. Maybe that's the point to make it harder to make those complicated scripts in something other than shell. In other languages it may be easier to make correct pipelines, but with heavy boilerplate. Then it's probably easier to count on libraries instead of pipelines, even if those libraries spawn processes in background.

I'm hoping for oil shell more as it means to address all those issues and more with an escape hatch to convert old scripts. Although I'm not so sure about the implementation details. For those in the know it made more sense to me after I read that OVM itself would become a VM for oil. So that there would be a single VM to run and not two as it is now. At least that's how I understand it.

POSIX shell is kinda magical, because it's nominally portable. However the rabbit seems to have died in the hat, because it's really that old. Shell was conceived in a different era with different limitations. I think that we should go further.

[0] take-in-and-out <(process-out) >(filter-this | grep that) | grep msg

In my experience, the parser definitely needs to be a library, but I'm not sure about the runtime.

There are a few places in POSIX where the parser has to be invoked recursively:

- command sub: $() and ``

- eval

- alias expansion. (I found some divergence in how shells implement this, but it does involve the parser.)

In Oil I also used the parser as a library in several other places:

- For interactive completion. Bash does not do this, and I don't believe any other POSIX-ish shell does. I wrote a bit about that in the latest blog post [1]. This turned out really well.

- For history expansion, because unevaluated words have to be picked out of previous command lines. Bash does not do this either.

Consider:

    $ echo ${x:-a b c}
    a b c
    $ echo !$
    echo c}
    c}

IMO this is fairly nonsensical behavior, and the underlying cause is that bash chooses to write duplicate, ad hoc parsers for its own language! There are many cases like this with completion, e.g.

    $ if ec<TAB>
    $ for i in 1 2 3; do ec<TAB>

Bash isn't smart enough to complete "echo" in these cases, because it doesn't know it's in the "first word' state.

It also chooses to treat = and : as completion word delimiters, even though they don't delimit normal words, and this causes a lot of problems that the bash-completion project patches over in a very ugly fashion.

----

As for the runtime, one problem is that the shell inherently modifies global process state. So there is a limit to the abstraction you can provide over it. For example consider this program:

    { echo hi
      ls / 
      echo bye
    } > out.txt

There's essentially one way to implement this with Unix system calls, but you couldn't have two different interpreters running them concurrently in the same process, because the process FD tables would get stomped on. (i.e. my definition of library is that you can make multiple instances of it with different parameters.)

----

The general idea of a shell that conforms to POSIX but provides a better interactive experience is a good one. (That seems to be the feedback on the Fish Shell 3.0 thread on the front page).

Although it is a huge amount of work! I hope that I will be able to metaprogram / compile Oil into something more compact, but that's an open problem now :) It is shaping up to be a better interactive shell than I originally thought though. Treating the parser as a library was a big win.

(Among other reasons, it's not a library in bash because it uses many global variables.)

[1] http://www.oilshell.org/blog/2018/12/16.html