Hush, a modern shell scripting language

I agree. As a language which intends to replace shell scripts, I would show first that the syntax is pretty much the same in Hush. I would put this page first https://hush-shell.github.io/cmd/index.html

Yup. Most shell scripts simply execute commands, handle files. It should begin with more examples using find, grep, sed, and a for loop.

To add to this, an example using pipes would be great too. I assume it's done the normal Bash way within command blocks. But I have no idea whether it's supported, after all the point is to use a different, more maintainable language.

But otherwise I think it's great, it has all the basics I'd expect from a modern language and still seems simple enough for quick scripts. I'll try it alone for the better syntax and the imho neat idea to return stdout & stderr separately in a dict.

I couldn't agree with this statement more. As someone who has developed their own shell over the last decade I've been keeping a close eye on what other people have been building too. So often I see people writing shells that are inspired by programming languages so their syntax looks amazing in documents. But they always strike me as being hugely tedious for repetitive and often quite dull tasks. The kind of 5 minute jobs that you drop into an interactive REPL to solve an immediate problem. And then maybe save as a shell script later if you find you're doing that task frequently.

People look at the problems with error handling and maintainability (which, as you highlighted yourself, are very real) but they end up throwing the baby out with the bath water when trying to solve those problems.

Ultimately it doesn't matter how well a shell is for scripting if it still creates more trouble as an interactive shell compared to (for example) Bash. I say that because we already have Python, Ruby, Perl, node.js, Jua, and a plethora of other languages that can be used for scripting. We don't need more scripting languages. We need better shells.

Yes, Python comes with 100% overhead from Shell. I have packages inside either the system or virtual environment tools. Honestly I err toward sh/bash because I write it, ship it, and forget it!

>the reason shell scripts have endured is precisely because they are extremely good at that

It's very easy to make something extremely good at this and much less error prone. That's the main problem.

>often imposing awkward control flow on you or making things that should be simple much much more complicated.

There's literally one primitive paradigm for control flow, both bash and other languages incorporate it. if... else. <----That''s it.

The weakness of the shell is that it did not emerge as an LR-parsed language that could easily be expressed with a concise yacc grammar. The "one true awk," for example, bundled a yacc grammar as part of the build until relatively recently. The complex grammar of the POSIX shell is ambiguous in the extreme in many situations.

The shell language was further constrained by POSIX that removed much Korn shell functionality, even though a public domain implementation of Korn existed. This was done to allow the shell parser to be very small, and Debian's Almquist shell compiles to under 100k on i386 (unlike bash, which trails it also in speed).

This new Hush shell looks a bit too wordy and reminiscent of javascript.

Anyone designing a new shell grammar should also design in such a way that Busybox could bundle this new interpreter, should popularity justify it.

Yes, exactly. I have a few slogans for this, but one is that the shell language should compose with the rest of the operating system, which is "processes and files". It's a situated language.

So I have been making the point that the "narrow waist" of shell is still processes and files. Whereas the narrow waist of Python and Lua is functions and compound data structures like dicts/lists (tables).

One thing that is not quite obvious until you design AND USE a shell language is that if you have these conflicting narrow waists, you have problems of composition, and you will offload this complexity onto your users.

From a very quick look at the hush docs, it is Lua with some shell-like stuff grafted on. I believe that will cause cause such problems in programs. They will be longer and harder to write. You will have more edge cases.

----

I hit this issue many times in Oil, and I wrote many posts tagged #software-architecture about it:

http://www.oilshell.org/blog/tags.html?tag=software-architec...

Unix Shell: Philosophy, Design, and FAQs http://www.oilshell.org/blog/2021/01/philosophy-design.html

makes several of these points.

This recent post

http://www.oilshell.org/blog/2022/03/backlog-arch.html

asks:

    - Should shells have two tiers?
      - Both external processes and internal "functions"? 
      - Both pipelines of bytes and pipelines of structured data?
    - Another one I'm working on now: Both exit codes and Python-like exceptions?

In Oil the answer is yes for the first 2, but a subtle point is that the former are PRIMARY and still the narrow waist; the latter are helpers. Dicts and Lists exist to help you deal with files (and produce JSON and TSV); but they're not the primary structures of the program.

I've also discussed this issue pretty extensively with the authors of the NGS and Elvish shells.

A related recent comment: https://old.reddit.com/r/ProgrammingLanguages/comments/ub5vi...

Having unsigned / signed and sync / async (what color is your function?) are similar problems of composition in language design. I call them "Perlis-Thompson" problems.

There is no easy answer but IMO it's vital to explicitly think about and address these problems. C has a flawed solution to the combinatorial explosion (implicit conversions), and it's caused problems and been lamented for ~50 years. There's also a good quote in that thread about why unsigned sizes in STL were a language design mistake.

I want a shell language that wonderfully encompasses both interactive usage and the creation of easy, safe, effective shell scripts.

I've come to the conclusion I can want that all I want, but it's not possible. Shell scripts and interactive usage have too many fundamentally opposing forces in them for one language to bridge the gap. You can be great at one or the other, and you can be bad at both, but you can't be great at both. There simply isn't such a thing, even in theory.

Closest you could get is a language that has two very closely related dialects, but the "dialects" aren't going to be "the same thing, but with a couple of little changes"; there's going to be a lot of differences, enough that people are going to be arguing seriously and with some merit that it's just two languages tied together at the hip.

Agreed. The problem with bash & Co. (if that's a problem) is that it's the result of a process of adding clunk after clunk of stuff in the only few ways that wouldn't break the language. So we have things like i=$(( i + 1 )) and [ $? -eq 0 ] and [ "$?" = "0" ]. Add to it the differences between "$@" "$" and $ and a number of similar things. Apart from that it's very good at what it does. Hush seems to start from a normal language like Python or Ruby and add the pipelining / redirection stuff which is usually very verbose and error prone to do in those kind of languages.

The point is that you can do things like create external processes, pipe them together, redirect output to files, with the same ultra-lightweight syntax of bash etc. Compare that to all the nonsense you have to do in Node or Python to pipe two processes together!

System level stuff sucks in Python. Dealing with files, I/O, permissions, etc is a real pain. It easily takes 5x as long and as many loc to do the same thing as in bash.

I can see the benefit of dropping to a command block to, say, run a command and filter the output with some | grep | awk | sort of whatever, and then seamlessly come back up to a more fully featured language to deal with that data.

Counterpoint: I work at the REPL of scripting languages roughly as much as I do the shell prompt.

I love Python, but if all I need to do is grab one part of one line of something, and put it into some other command, then I'm just going to use some unholy mixture of sed, awk, or whatever. If I did the same thing in Python, I'd end up with thirty lines or more.

The purpose of OP's project kind of reminded me of shell.js (shx) [1] which is a nodejs library that wraps all kinds of common UNIX commands to their own synchronously executed methods.

I guess that most shell projects start off as wanting to be a cross-platform solution to other operating systems, but somewhere in between either escalate to being their own programming language (like all the powershell revamps) or trying to reinvent the backwards-compatibility approach and/or POSIX standards (e.g. oil shell or zsh).

What I miss among all these new shell projects is a common standardization effort like sh/dash/bash/etc did back in the days. Without creating something like POSIX that also works on Windows and MacOS, all these shell efforts remain being only toy projects of developers without the possibility that they could actually replace the native shells of other operating systems.

Most projects in the node.js area I've seen migrate their build scripts at some point to node.js, because maintaining packages and runtimes on Windows is a major shitshow. node.js has the benefit (compared to other environments) that it's a single .exe that you have to copy somewhere and then you're set to go.

When I compare that with python, for example, it is super hard to integrate. All the anaconda- or python-based bundles for ML engineers are pretty messed up environments on Windows; and nobody actually knows where their site-packages/libraries are really coming from and how to even update them correctly with upstream.

[1] https://github.com/shelljs/shelljs

> The only way to run these scripts is to install the hush interpreter and run the script through it

Exactly like any other shell, and consider how inconvenient and hazardous a transpiler "to a common shell script" would be.

Yes, this is the first thing that struck me. I don't know why people invest so much time inventing new languages and then fail at the last mile.

If the front page of a new lang site doesn't have language examples showing not just hello world but core differentiating use-cases, why should I bother?

Frankly, I think that only, say, three things are worth keeping from traditional shells: pipes, redirections, and $-variables.

Most of the rest in them are design choices dictated by highly constrained machine resources and, most of all, incremental evolution. I'd say that Bash as a language is larger than Hush, while being conceptually much less clean, and in many regards less ergonomic.

> It suggests that "2>1" and "2> 1" have very different results

FYI, "2>1" and "2 >1" already have very different results in Bourne shell. So I don't think it's an unforgivable sin.

attacking the problem of improving shell scripting while attempting to keep the syntax

(Oil author here) Slight correction, The project has two parts

(1) OSH (bin/osh) will run your existing scripts, so in that sense it does keep the syntax

(2) The Oil language (bin/oil) is an upgrade from OSH, but it can also be thought of as a clean slate language:

https://www.oilshell.org/release/latest/doc/oil-language-tou...

It's really the same interpreter with a few parse time and runtime options. There were surprisingly few compromises (probably the biggest one is the shell-like string literal syntax, which I mention.)

The biggest changes are that we take over ( ) and { } , so you can write stuff like:

    if (x > 0) {
      echo 'positive' | tac
    }

Rather than

    if test "$x" -gt 0; then
      echo 'positive' | tac
    fi

We add a Python-like expression language between () and surprisingly few things need to change to make it feel like a "new language".

Thanks for mentioning Oil :)

This isn't really true at all. We have an ungodly amount of operational process and knowledge tied up in bash scripts deployed across our fleet, but we could easily deploy a different shell, and use that. We don't use bash because it's the least common denominator on our systems --- in fact, I think in an era of AMIs and Dockerfiles, few people do. We use it because of path dependance.

I think part of the problem is that we still don't have a standard /bin/sh even though there is a written standard. It feels fair to me to suggest that you can call dash the working standard on Linux, but you can't trust that beyond Linux systems(and I'm ignoring installs that are busybox-based here too).

And that also feels like the problem with extending a script with a few bashisms too, as you're suddenly trying to remember when that feature appeared. Off the top of my head I'm wondering what things won't be available to Mac users because it ships with a pre-GPLv3 bash for example.

It wasn't that long ago that a fresh Unix install of any flavor didn't come with Perl, Python, zsh, or other common shell or scripting languages. And there's already a trend away from installing those by default. Since adding them now takes mere seconds, the barrier is low.

How often do you run into an issue where things only would have worked if you use sh? What domain are you working in?

I dunno, I think the future of interesting shell scripting languages can either be a.) the basis for a new distro, which will decide to use it for the system scripts, b.) be intended for use with containers in some manner and can simply be added as another dependency, or c.) be primarily intended to be used interactively in some unique way, so it'll just be installed by individuals.

There will always be a place for the universal shells and text editors. Bash and vim aren't going away any time soon. But there are plenty of places for non-default choices. Vim's existence doesn't mean Sublime doesn't have a place.

Oh my god, your first point. I had to pass arbitrary arguments through a couple layers of shell and I couldn't do it. Gave up and rewrote it all in Python, even though it's annoying in other ways, and I'm decent at shell scripting. It's a nightmare.

Most of those features are in PowerShell, especially version 7.

It uses strict ('reliable') argument passing, strong typing, etc..

It has a bazillion independent streams such as "Debug", "Verbose", "Warning", "Error", "Progress", "Output", "Information", etc...

It has "foreach -parallel", which is a fun way to make the CPU fan spin up.

> "Recurse through this directory, while ignoring .git and vim backup files". Read this file into an array, splitting by newline, in a single line of code. It's tiresome to implement that kind of thing in every script I write. At the very least it should be simple and comfortable.

    $lines = dir -Recurse -Exclude *.git, *.vim | Get-Content

Not exactly rocket science in most shells? In most shells you can make this a modular function that takes in a list of file names, but PowerShell's "dir" doesn't output file names. It outputs file objects, so then it is just one step to do a filter on something like extension (or length, or whatever):

    dir | ? Length -lt 1MB | ? Extension -in @( '.cs', '.rs', '.cpp' )

In PowerShell you can create functions that integrate with the object-oriented pipeline. You can pipe full objects into a function, and objects out. Object attributes can be mapped automatically to function arguments too. You can have begin/process/end blocks so you can do things like open a DB connection, pipe in data, then close the connection neatly without needing to do this all manually each time. Literally something as trivial as this:

    Get-Stuff | Out-DatabaseTable -TableName 'stuff_log' -Server '...'

Seriously. Give it a go. It works on Linux too, and it's more unix than unix.

Bash is definitely painful to write - I write shell scripts a lot and I enjoy it (in a masochistic kind of way), and I wholeheartedly agree with most of the criticisms you've written. However, the greatest strength of Bash is its ubiquity - it is available on almost every modern Unix-like environment - and when combined with standard POSIX tools it can provide a-little-less-painful environment for writing reliable shell scripts.

In order to prove it, I will do my best to provide some kind of solution to each of the problems you have mentioned. Note that they are in no way perfect solutions, and all of them are just hacks that work around Bash's inherent clumsiness.

---

> 100% reliable argument passing

Wrapping a variable in double-quotes does this:

    var="1 2 3 4"
    for x in "$var"; do echo $x; done

For double-parsing (e.g. SSH commands) you can wrap the entire expression in single-quotes to maintain the original string until the second shell:

    ssh user@host 'var="1 2 3 4"; for x in "$var"; do echo $x; done'

> 100% reliable file iteration

The "find" command does exactly this, and it works for whatever weird characters in the filename you have.

    find ./directory/ -type f -exec SOME_COMMAND {} \;

> No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long.

Do not evaluate arguments directly in shell, use xargs to feed the arguments as standard input. E.g.:

    find ./directory/ -type f | xargs SOME_COMMAND

> Excellent path parsing. Such as filename, basename, canonicalization, finding the file extension and "find the relative path between A and B".

    dirname /foo/bar
    basename /foo/bar
    realpath ./symlink/to/foo/bar
    basename /foo/bar | sed 's/.*\.//'
    realpath --relative-to ./dirA/fileA ./dirB/fileB

> Good error handling and reporting

This is a tough one. Good error handling/reporting in Bash is extremely hard and requires discipline, but it's possible.

To exit immediately as soon as a command exits with non-zero exitcode:

    set -e

However, this does not play nice with temporary resources, so I personally prefer hand-written error checks:

    SOME_COMMAND || { echo "Some Command has failed" && exit 1; }

Error reporting depends on a scenario, but I generally have a pattern of doing checks like:

    if [ "$var" != "expected" ]
    then echo "Error: var different from expected" && exit 1
    fi

And it works fine most of the time.

> Easy capture of stdout and stderr, at the same time. Either together or individually, as needed.

To capture both stdout and stderr in a same stream:

    SOME_COMMAND |& SOME_FILTER

Capturing stdout and stderr in different streams is tough. The easiest way I can think of is to use a pipe and job control mechanisms:

    mkfifo pipe
    SOME_COMMAND 2>pipe | FILTER_1 &
    < pipe FILTER_2 &
    wait
    rm pipe

* Excellent process management. We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available, while being able to report progress, record each failure, and abort the entire thing if needed.

While bash is truly terrible when it comes to concurrency, I find GNU Parallel to be pretty satisfying for most concurrent shell-scripting:

    cat FILE.txt | parallel --timeout=30 SOME_COMMAND {} \|\| echo "Some Command failed with argument: {}"

For POSIX purity, xargs also can be used with "-L 1" argument that parses a single line per command iteration. For parallelism, there is also a "--max-procs" argument.

* Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere".

I find that adding "set -x" to the top of your shell, which prints each command with "+" prefix as it is expanded and executed, very useful for error tracking.

* Give me helpers for common situations. Eg, "Recurse through this directory, while ignoring .git and vim backup files". Read this file into an array, splitting by newline, in a single line of code. It's tiresome to implement that kind of thing in every script I write. At the very least it should be simple and comfortable.

There are helpers out there for most common situations - it's just that they are implemented as CLI tools, and not officially part of the shell. And in the scenario where you can't expect the availability of those unofficial tools, you can always write your own small library of commonly used Bash functions, and just copy-paste them into your script whenever you need them. It's ugly, but still possible.

In my (biased) opinion Perl comes close to ideal with judicious use of Path::Tiny and Capture::Tiny.

Shameless plug, but the problems you've described are exactly the problems I was looking to solve with my own shell (https://murex.rocks / https://github.com/lmorg/murex)

> 100% reliable argument passing. That is, when I run `system("git", "clone", $url);` in Perl, I know with exact precision what arguments Git is going to get, and that no matter what weirdness $url contains, it'll be passed down as a single argument. Heck, make that mandatory.

Variables are parsed as tokens so they're passed to the parameters whole, regardless of whether they contain white space or not. So you can still use the Bash terseness of parameters (eg `git clone $url`) but $url works in the same way as `system("git", "clone", $url);` in Perl.

> 100% reliable file iteration. I want to do a "for each file in this directory" in a manner that doesn't ever run into trouble with spaces, newlines or unusual characters.

Murex is inspired by Perl in that $ is a scalar and @ is an array. So if you use `f +f` builtin (to return a list of files), it returns as a JSON array. From there you can use @ to expand that array with each value being a new parameter, eg

  rm -v @{ f +f } # delete all files

or use $ to pass the entire array as a JSON string. Eg

  echo ${ f +f }
  # you could just run `f +f` without `echo` to get the list.
  # This is just a contrived example of passing the array as a string.

Additionally there are lots of tools that are natively aware of arrays and will operate on them. Much like how `sed`, `grep`, `sort` et al treat documents as lists, murex will support lists as JSON arrays. Or arrays in YAML, TOML, S-expressions, etc.

So you can have code that looks like this:

  f +f | foreach file { echo $file }

> No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long.

This is a kernel limit. I don't think there is any way to overcome this without using iteration instead (and suffering the performance impact form that).

> Excellent path parsing. Such as filename, basename, canonicalization, finding the file extension and "find the relative path between A and B".

There are a number of ways to query files and paths in murex:

- f: This returns files based on meta data. So will pull files, or directories, or symlinks, etc depending on the flags you pass. eg `+f` would include files. `+d` would be include directories. or `-s` would exclude symlinks.

- g: Globbing. Basically `*` and `?`. This can run as a function rather than being auto-expanded. eg `g *.txt` or `rm -v @{ g *.txt }`

- rx: Like globbing but using regexp. eg `rx '\.txt$'` or `rm -v @{ rx '\.txt$' }` - this example looks terrible but using rx does sometimes come in handy if you have more complex patterns than a standard glob could support. eg `rm -v @{rx '\.(txt|rtf|md|doc|docx)$'}`

The interactive shell also have an fzf like integration built in. So you can hit ctrl+f and then type a regexp pattern to filter the results. This means if you need to navigate through complex source tree (for example) to a specific file (eg ./src/modules/example/main.c) you could just type `vi ^fex.*main` and you'd automatically filter to that result. Or even just type `main` and only see the files in that tree with `main` in their name.

> Good error handling and reporting / Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere".

A lot of work here.

- `try` / `trypipe` blocks supported

- `if` and `while` blocks check the exit code. So you can do

  if { which foobar} else {
    err "foobar does not exist!"
  }

  # 'then' and 'else' are optional keywords for readability in scripting. So a one liner could read:
  # !if {which foobar} {err foobar does not exist!}

- built in support for unit tests

- built in support for watches (IDE feature where you can watch the state of a variable)

- unset variables error by default

- empty arrays will error by default when passed as parameters in the next release (hopefully coming this week)

- STDERR is highlighted red by default (can be disabled if that's not to your tastes) so you can clearly see any errors if they're muddled inside STDOUT.

- non zero exit numbers automatically raise an error. All errors return a line and cell number so you can find the exact source of the error in any scripts

Plus lots of other stuff that's referenced in the documents

> Easy capture of stdout and stderr, at the same time. Either together or individually, as needed.

Murex handles named pipes a little differently, they're passed as parameters inside triangle brackets, <>. STDERR is referenced with a proceeding exclamation mark. eg a normal command will appear internally as:

  command1 <out> <!err> parameter1 parameter2 etc | command2 <out> <!err> parameter1 parameter2 etc

(You don't need to specify <out> nor <!err> for normal operation).

So if you want to send STDERR off somewhere for later processing you could create a new named pipe. eg

  pipe example # creates a new pipe called "example"
  command1 <!example> parameter1 parameter2 | command2 <!null> parameter1 parameter2

This would say "capture the STDERR of the first command1 and send it to a new pipe, but dump the STDERR of command2".

You can then later query the named pipe:

  <example> | grep "error message"

> Give me helpers for common situations. Eg, "Recurse through this directory, while ignoring .git and vim backup files". Read this file into an array, splitting by newline, in a single line of code. It's tiresome to implement that kind of thing in every script I write. At the very least it should be simple and comfortable.

This is where the typed pipelines of murex come into their own. It's like using `jq` but builtin the shell itself and works transparently with multiple different document types. There's also an extensive library of builtins for common problems, eg `jsplit` will read STDIN and output an array split based on a regexp pattern. So your example would be:

  cat file | jsplit \n

> That's the kind of thing I care about for shell scripting. A better syntax is nice, but actually getting stuff done without having to work around gotchas and issues is what I'm looking for.

I completely agree. I expect this shell I've created to be pretty niche and not to everyone's tastes. But I'd written it because I wanted to be a more productive sysadmin ~10 years ago and since I've moved into DevOps I've found it invaluable. It's been my primary shell for around 5 years and every time I run into a situation where I'm like "I wish I could do this easily" I just add it. The fact that it's a typed pipeline makes it really easy to add context aware features, like SQL query support against CSV files.

> 100% reliable file iteration. I want to do a "for each file in this directory" in a manner that doesn't ever run into trouble with spaces, newlines or unusual characters.

In UNIX file names, there is exactly one unusual character: ASCII NULL (or '\0'). Every other character is usual, including spaces, newlines, tabs and other historical ASCII control characters.

------

> No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long.

Shell does not impose length limits, the UNIX kernel does by defining the limit of how much can be passed into the «execve» syscall which UNIX shells use to create new processes. You can find the length limit on your system by running «getconf ARG_MAX» in the shell as it varies across different systems (it varies even across different versions of the same system; in Linux, it is now reportedly 1/4th of ulimit -s). The command line limit is calculated using following constituents:

  command line limit = «environment size (env | wc -c)» *plus* «(ARG_MAX *minus* environment size» *minus* «POSIX recommended 2048»).

How the length limit works is explained at length (please pardon the pun) here: https://www.in-ulm.de/~mascheck/various/argmax/

------

> Excellent path parsing. […] finding the file extension […]

File names in UNIX do not have extensions, they simply have names. Is «.bashrc»: 1) a full and complete file name or 2) an empty file name with the «bashrc» extension? It is (1). Moreover, any valid character can be used as a separator and its interpretation is either left out (almost always), or the interpretation is left up to the semantically aware app. One is free to use a comma or the Javanese wasana pada as part of the extension; that is, «file,exec» and «file꧅ ꦆ ꧅exec» are both valid and both have the extension of «exec» – as long as the file system supports the appropriate character set. This is also why «basename», to work correctly, requires the separator as part of the imaginary «extension», i.e. «basename myfile,exec ,exec» will always give «myfile» as the result.

DOS style extensions are a made up convention that neither the kernel, nor the shell, nor file processing utilities enforce as they are file extension unaware. It is better to think of UNIX file names as being made up a prefix and an optional, arbitrary length suffix (but not an extension).

------

> Excellent process management. We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available

With respect to «every core available». You almost certainly don't want that and should overprovision the number of running processes compared to the number of cores on your system.

UNIX has supported multiprocessor systems for a very long time. Despite massive improvements in the hardware performance, disks and networks are still the slowest moving (or still) parts. They were even slower when UNIX was in its relative infancy when the CPU time was also very expensive. Therefore, the CPU time had to utilised efficiently whilst waiting for a disk to return a string of bytes.

File processing tasks spend their time between: 1) waiting for I/O (in the blocked state) and 2) actually processing (the running state). Since the disk is still very slow compared to the speed of a modern CPU, the UNIX process scheduler blocks the process until I/O completes and checks the process run queue in the kernel to see if there is another process ready to compute something (i.e. in the running state). This inherent interleaving of «blocked for I/O» and «running» process states can be used to an advantage depending on a few factors.

The less is size of the unit of data an app processes and the larger is the total size of the input (i.e. the input and output files), the more time the app spends in the «blocked for I/O» state, and most of the CPU time is simply wasted unless there is something else to do. But if we know such specifics (the size of the unit of work and the size of the input), we can overprovision the number of processes thereby utilising the CPU compute time more efficiently whilst the disk controller is transferring bytes into the memory via the direct memory access (I am oversimplifying a few bits here). This is the reason why «make -j12» will compile almost always faster than «make -j8» on a 8 CPU core system on projects with a large number of small(-er) files – because of the I/O overhead. Whilst there is no universal formula, 1.5x process overprovisioning factor is a decent starting point. For simpler daily file processing tasks GNU parallel is good enough to spare oneself of headaches of such computations, though.

Therefore, the «We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available […]» does not make sense in the context of UNIX shell languages and the process scheduling in UNIX. In fact, you will underutilise your 128 CPU cores, sometimes pretty heavily, unless you correctly account for the I/O factor.

Only if the app/process is aware of how to efficiently parallelise its own workload (because it knows its workload better than anyone else), then and only then it does need to know how many cores there are available at its disposal. No scripting language / shell can solve this problem.

------

> Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere".

The excellent error reporting has existed in UNIX since day 1. It is called the process status code; anything that is not a zero status code indicates an error. The semantic interpretation of each speific numeric status code, however, is entirely decoupled from the thing that might fail and is documented in the man page. The status code of 3 in «mv» and in «git» will mean two completely different things, therefore the specific status code processing is localised to the process invocation point in shell scripts. Most of the time, though, I personally don't want my shell script to explode with an error message from a random failed command unless it is something of extreme importance to me; checking for the process exit code and acting upon it accordingly is sufficient and is good enough.

Whether such an approach is a good thing or not is a matter of a debate. Global lists of errors and/or error messages require an official register of both diligently maintained and updates of which to be centrally coordinated, which I don't think could work with the open source.

Other operating systems have attempted to mandate error codes with complex structures and a well defined (and sometimes written by a professional technical writer!) error message. Yet, they have had limited success. Yes, OS/400 running on an AS/400 could inspect a failed process' error code and automatically dispatch a message to an IBM service centre to order a spare part for a specific failing piece of hardware whose SKU would have been deduced from a specic part of the error code without requiring the human intervention, but that is somewhat of an extreme and extravagant example and is certainly not mainstream.*

EDIT: «make -j12» vs «make -j8» explanation.

For access to libraries, and familiarity to programmers, this is probably the best shell replacement.

The reality is that blame shaming devs for not learning something isn't effective. No matter your personal beliefs, nobody is getting fired for not knowing the intricacies of sed or bash or awk. The only options are keeping archaic artifacts around and have the elders maintain it, or improve it for everyone. I have tremendous respect for our unix/posix legacy but at the same time they didn't get everything right. How could they?

That said, it's harder to build consensus today. Currently people are replacing shell scripts with all kinds of stuff, and that's not good either. We need something good enough and ubiquitous, imo.

I am a greybeard, old school kinda unix guy, but at some point insisting everybody learn latin to understand the scriptures is just not going to work.

As long as Powershell dies I'm more than happy to agree that any bash script that exceeds 24 lines should probably be written in something newer.

As you can probably guess from my username, I am among the last people who would disagree on the usefulness of common shell utils, and the benefit of knowing them.

But I still hold that one should rewrite into a proper programming language, as soon as a shell script becomes too large, and that point is usually around the 100 LOC mark. Yes, I can do amazing things just with sh/bash and the coreutils. But a small Go program or python script, is just alot more readable, at least imho, because they impose STRUCTURE, which the shell lacks on purpose: It's a CLI first, and a programming language second.

I can read through several hundred lines of Go in one sitting. When asked with analyzing some old bash script, and `wc -l` tells me something along the lines of 400+, I grab the coffee machine and bring it to my desk.

The section on the try operator¹ includes an example showing the syntactic sugar to reduce the verbosity. It does feel like it might be better to introduce it earlier on, as I was feeling the same way as you until I saw that.

¹ https://hush-shell.github.io/error-handling.html#try-operato...

Destructuring and a little bit of pattern matching could go a long way here.

Side note: The most straightforward way to incoporate YAML as an elevated text "type" would be to define a new type of pipeable file descriptor on all commands, specifically meant for replacing shell script options / flags. (That way options aren't mixed with stdin)

Edit: and you get typing validation for free by running your yaml options through a yaml schema validator should you choose to do so

You might love autosetup, a Tcl-based autoconf tool chain replacement.

Not just Tcl-based, but designed to work with a simple Tcl subset shell called Jim, which compiles from one source code file. So you can bootstrap your project in a mostly-platform-independent way with a standard C compiler as the only prerequisite.

https://msteveb.github.io/autosetup/

Thanks for the resource. I have been using Clojure and Babashka to bring the feeling of functional programming to the shell but have not been happy so far.

This looks really cool! Is it your daily driver?

Yes, and it lives on in a variety of minimalist environments under active development, such as:

* U-boot: https://github.com/u-boot/u-boot/blob/master/common/cli_hush...

* Busybox: https://github.com/brgl/busybox/blob/master/shell/hush.c

* Barebox: https://github.com/barebox/barebox/blob/master/common/hush.c

Yes, I find it odd how there's a massive blind spot when it comes to powershell.

Proponents of strong typing everywhere in their programming languages are put off by a shell that dares pipe objects rather than having everything as strings.

I get that Unix has a long tradition and it's hard to change, but powershell is genuinely a modern shell that while uncomfortably verbose without aliases is extremely powerful and feels modern.

I've already opened a pull request for the latter https://github.com/hush-shell/hush-shell.github.io/pull/1

Wouldn't the value of the fifth index be executed there, which is false and would hence fail?

Not sure either though

> safe_div_mod and sometimes safe_division

Aren't those different things? Save division wouldn't use mod by default, right? The function for save division is missing though, but it might just be a built-in

A recent one is Zuo, a tiny Racket for scripting/makefiles: https://docs.racket-lang.org/zuo/index.html

I'm partial to zx for that (it has optional typescript support) and pretty reasonable ergonomics

This is a dynamically typed language. Null makes sense there.

It also makes sense in a statically typed language with union types, think `string | undefined` in TypeScript. The extra layer added by Some/None is not fundamentally necessary for safety.