Advanced Shell Scripting with Bash (2006) [pdf]
uniforumchicago.org
uniforumchicago.org
I think I've finally settled on Lua- It's small, coherent, surprisingly fast (especially with LuaJIT) and has the "virtually no startup cost" I was looking for (which ruled out, for example, Elixir for a lot of things).
And, I've finally figured out how to bang my nix-darwin flake.nix config in a way that gets it AND a bunch of popular libraries installed together and all seeing each other.
Wondering what others have found to address this. Or if they just stuck with Bash, since it's just not going away anytime soon.
It has similar aims to https://www.nushell.sh/
I've actually already written a lot of the functionality I see here as bash (and other) functions, lol (things like log, filter, debug, retry, repeat, map, fetch (a URL, with retry)... etc.). (things like: enumeration functions, table pretty-printing, etc.)
> I've actually already written a lot of the functionality I see here as bash (and other) functions
That's exactly the reason these are part of stdlib. No reason for everybody to reimplemnt these with slight variations and copy it around.
Perl was originally written as an amalgamation of grep,sed,tr, awk and I am sure a few more unix utilities with their own mini languages. The idea was to use one language instead of tying together half a dozen mini-languages in a shell script. And it worked really well. Perl being a demon with text munging didn't hurt.
Raku keeps this heritage but adds so much more (for better or worse :-) ). It inherits ideas from Lisp and functional programming languages. The thing that impressed me was, how easy it was to use concurrency.
I was aware of it, but not familiar until I looked at the link you shared. Looks like a Scheme, which for the uninitiated is a different approach for shell scripts.
It's consistent, "infinitely" terse (you can create dsl-like apis for your problem domains), proper programming language.
First time I used it for scripting was when I had to process some xmls and do bunch of processing and gluing together – I was impressed how quick and easy it was without knowing anything about it and how readable the whole thing ended up being.
[0] https://www.gnu.org/software/guile/manual/html_node/Scriptin...
The “Unix way” stands on the shoulders of stdin, stdout, pipe, and fork.
And the shells make that a first class concept. Breaking up the command line and linking processes together is the #1 job of a shell, everything else is gravy.
The scheme and lisp shell takes fail here, just because you have to go through the gymnastics of syntax.
My fantasy would be to have something like this:
(! ls | (myfunc) | sort -r -n | >list)
(“100 x.txt” “200 y.txt”)
Piping system commands into my own functions within the environment.Sure, I could do:
ls | myfunc.scm | sort -r -n
But then I can do that with anything, not just scheme (there’s that Unix superpower again). $ txr
This is the TXR Lisp interactive listener of TXR 299.
Quit with :quit or Ctrl-D on an empty line. Ctrl-X ? for cheatsheet.
Poke a few holes in TXR with a fork before heating in the microwave.
1> (flow (glob "/proc/*") (take 10))
("/proc/1" "/proc/10" "/proc/101" "/proc/102" "/proc/103" "/proc/104"
"/proc/107" "/proc/1074" "/proc/1086" "/proc/11")
2> (flow (glob "/proc/*") (take 10) (map (op trim-left "/proc/")))
("1" "10" "101" "102" "103" "104" "107" "1074" "1086" "11")
3> (flow (glob "/proc/*") (take 10) (map (op trim-left "/proc/")) (sort @1 : toint))
("1" "10" "11" "101" "102" "103" "104" "107" "1074" "1086")
4> (flow (glob "/proc/*") (take 10) (map (op trim-left "/proc/")) (sort @1 greater toint))
("1086" "1074" "107" "104" "103" "102" "101" "11" "10" "1")
How about we parse digit sequences out of the path names and combine them with the originals, then sort on those digit sequences turned into integers: 5> (defun get-nums (str)
(flow str (tok #/\d+/) (map toint)))
get-nums
6> (flow (glob "/proc/*") (take 20) (csort @1 greater get-nums))
("/proc/11170" "/proc/11169" "/proc/1222" "/proc/1199" "/proc/1198"
"/proc/1194" "/proc/1133" "/proc/1104" "/proc/1086" "/proc/1074"
"/proc/117" "/proc/107" "/proc/104" "/proc/103" "/proc/102" "/proc/101"
"/proc/12" "/proc/11" "/proc/10" "/proc/1")
csort is caching sort: it caches the result of the key function through which the elements are projected to form the argument of the comparison.
In other words, so we don't wastefully call get-nums multiple times for the same path.I would be something like this,
(->
ls
myfunc
sort -r -n
(> list))
Followed by "Select => Execute" with the mouse, or "Execute last form" with keyboard shorcut.One that strove to do both WOULD have to do all the things you suggested as well (job management, stdout/stderr redirection etc.) So options like oil shell, nushell, fish etc.
I was just talking about "a commandline tooling language" in general, as far as use-cases. And for that, Lua (and Moonscript, which compiles to Lua but is way nicer and has been featured on HN a few times over the years) seems to fit the bill (1-based indexing aside, ugh)
I'm one of those folks that still can't do LISPs though. If there was a way to replace some or all of the parens with significant indentation instead (which I believe Racket makes possible... but that only works in Racket), I might reconsider it.
With structural editing, no one ever have to match parens or count parens at the end...
YMMV ofcourse.
You have to reinvent the wheel every other day since the standard library doesn't come with much included
Or just text-processing things, where it's most suitable?
(I think Awk is underrated, for sure!)
(The intro page may be a bit misleading. You can freely mix-and-match existing, unstructured as well as nushell-built-in structured commands in the pipeline, as long as you convert to/from string streams - its not mandatory to use the structured built-ins. For example if an existing cli tool has json output, you can use `tool | from json` to turn it into structured data. There are also commands like `detect columns` that parses classic column output, and so on - the tools to mix-and-match structured and unstructured data are convenient and expressive)
Some highlights:
- automatic command line arguments and help by defining a main function and adding comments to each argument - e.g. https://github.com/nushell/nushell/discussions/11969
- run commands with controlled parallelism: https://www.nushell.sh/commands/docs/par-each.html
- easy parsing of raw input https://www.nushell.sh/commands/docs/parse.html
- support for a wide variety of data formats https://www.nushell.sh/commands/categories/formats.html
- built-in support for talking to SQLite databases: https://www.nushell.sh/book/loading_data.html#sqlite
edit: it looks like Mitchell Hashimoto was recently impressed too https://x.com/mitchellh/status/1907849319052386577 - rich functional programming library that blends with pipeline syntax https://www.nushell.sh/book/nushell_map_functional.html
Addendum: Its not my login shell. I run it ad-hoc as soon as the command pipeline i'm writing starts getting too complicated, or to write scripts (which of course can be run from outside nushell too, so long as they have the correct shebang)
See, FileUtils for things like rm_rf, rake provides sh which can execute commands, then there's FileList and Pathname. It even comes with its own template engine (erb) if you need that. Regexes as first class citizens are awesome for shell scripting.
I can also recommend using optprase + ostruct for commandline argument parsing.
Like ruby (I guess), python is very pleasant to work with for small scripts and it's portable between linux and windows, which is really cool (at least in my use cases)
In python, glob, Path and subprocess are mostly all I need...
If you're familiar with Python, give xonsh (https://xon.sh/) a go. It's a Bash-like shell but the syntax is Python. It has the same ease of writing shell scripts as Bash does, sans the insane language.
Been using it since 2018.
(I'm not particularly inclined to Python, unfortunately, due to the dependency issues and some personally negative opinions about its design... but I like the idea of Xonsh if you're a Python person! I wish I had something like that for Elixir!)
- Elvish https://elv.sh
- Murex https://murex.rocks
- Oil https://oils.pub
However the other shells are also highly polished. I’d definitely recommend each and all of them.
Executing a process and bringing back status, stdout and stderr seem tedious. While in Perl, it is as natural as it is in a bash script (well almost).
Raku on the other hand is _Fun_.
i do all my automation with one tiny 20 line raku script that manages a thousand things. hell, the entire monitoring site for our frog is basically just a pile of mixins with a neat little pipeline interface unifying everything from request profiling and html templating to reading from and filling that cache in parallel whenever it remotely pulls the sensor data off various things and gets the current webcam frame.
As a minor example, I hate having to remember (or explain) ${var##pattern}
It is useful for manipulating paths, but it is cryptic.
in comparison, something like:
newfile = os.path.join(newpath, os.path.basename(oldpath))
IMHO is more self-explanatory and robust.Also, argparse is the best.
https://mywiki.wooledge.org/BashGuide
Also, ShellCheck is an invaluable resource to use on all your scripts. When it complains about something, read up on why it doesn't like it and then decide to either fix your code or put in an exception to tell ShellCheck to ignore that particular issue for the next line.
(This statement primarily applies to existing stringly-typed scripting languages, which are nightmarish to maintain and debug. PowerShell, nushell, or similar solutions have a much higher complexity ceiling.)
Seat Belts and Airbags for bashI really want to reach for that pun and suggest: "Seat Belts and Airbags for bash safety" as an even better title.
> The core of the Tcl interpreter has been replaced with an on-the-fly compiler that translates Tcl scripts to byte codes; a new interpreter then executes the byte codes. In earlier versions of Tcl, strings were used as a universal representation; in Tcl 8.0 strings are replaced with Tcl_Obj structures ("objects") that can hold both a string value and an internal form such as a binary integer or compiled bytecodes.
http://www.ira.inaf.it/Computing/manuals/tcl/man-8.0/Changes...
http://www.ira.inaf.it/Computing/manuals/tcl/man-8.0/Changes...
IMO it applies to all languages. Fancy language features usually make for poor maintainability, e.g. metaprogramming.
Here is are the slides: http://uniforumchicago.org/slides/bash_2025-03-25.pdf
Here is the recording of the video: https://www.youtube.com/watch?v=DvDu8_A2uhs
Here is the stringent.sh library that is shown in the presentation: https://github.com/pottmi/stringent.sh
I will throw the pdf on my gitlab account that has the stringent.sh script.
The social graces having been observed, if you were to update this presentation to reflect modern bash, and especially reference variables, what would be your top 3 additions?
(I am a big fan of reference variables. They make it possible to, among other things, do some funky things that would otherwise require liberal use of eval, which I eschew. Figuring out how to ensure that the target of a reference variable was in fact an associative array while running under `set -u` was a trick of which I am proud. Note: what I did works and isn't totally ugly, there may be a better way, YMMV, IANAL, etc.)
Second Q, if you will permit: Given the memory leaks in associative array management that have been corrected over the last several years, what are your thoughts on a) whether or not bash should have ever gone that route, and, potentially more controversially, b) the current state of bash maintenance?
(I'm not going to ask about embedding multiple levels of ${...#...} or ${...#...}, because even if it worked, readability would tend to 0 very, very quickly. I'd rather be repetitive and readable and maintainable. Well. Most of the time.)
Thanks for the tip on reference variables. definitely better than eval. i will add that to the presentation.
I will look into using set -u and what I can do with that and add it to the presentation. Perhaps make a companion presentation on debugging bash.
Another good presentation would be on how to use bash safely with sudo. I have a sudo presentation that is pretty good for it is for sudo 1.7 and the current version is 1.9 so I would need to update it before giving it.
I have not used associative arrays in bash. I would probably tend to use python if i needed that capability so I am not likely to use aa in bash soon.
Regarding nesting levels of self modifying variables: Yeah, I tried to do it once and failed and did not try to figure out any tricks to get it to work. I just moved to multiline which is easier to debug and comment on for the poor bastard that has to fix my code.
From the archived site's abstract [2]:
>"bash (and scripting languages in general) act as the glue that hold other system components together. This presentation will focus on the under utilized features of bash that are critical to building production quality scripts. Demos will show you how and why to turn these features on."
EDIT: seems the website being down is a me issue (?), but still, having the archives doesn't hurt. ---
[1]: https://web.archive.org/web/20241212102046/https://uniforumc...
[2]: https://web.archive.org/web/20240719092417/http://uniforumch...
March 2025 presentation by the same author, "Seat belts and Airbags for bash"
slides: http://uniforumchicago.org/slides/bash_2025-03-25.pdf
video (87m): https://www.youtube.com/watch?v=DvDu8_A2uhs
See it here: https://github.com/pottmi/stringent.sh
from subprocess import run, PIPE
home = os.getenv("HOME")
try:
os.chdir(f"{home}/dev")
except FileNotFoundError:
sys.exit(1)
out = run(("git ls-remote https://github.com/%(repo)s refs/tags/%(tag)s"
% {"repo": shlex.quote("myorg/myrepo"), "tag": shlex.quote("v1.2.3+deadbeef")}).split(" "),
stdout=PIPE, stderr=PIPE)
The line noise is outrageous as compared to a language built for changing directories and running commandsThat's not even getting into the pty portion, where the progress of the output isn't visible until the end of it, which is terrible DX
If you consider more complex tasks, bash quickly gets unwieldy, and the situation is reversed.
Pythons has a lot of advantages, including full flexibility, error handling etc.
cd $HOME/media
git ls-remote ...
(Not sure about the equivalent of shlex.quote, but in the worst case, you can just use "from shlex import quote as q" or something).So yes, there are good alternatives to bash - even Python based.
[0] https://xon.sh/
repo = "myorg/myrepo"
tag = "v1.2.3+deadbeef"
out = check_output(["git", "ls-remote", f"https://github.com/{repo}", f"refs/tags/{tag}"], cwd=os.path.expanduser("~/dev"), stderr=PIPE)
Converting exceptions to exits is an anti-pattern that hides the source of the error, f-strings has been a thing for 10 years and argument quoting happens automatically. Progress of output isn't visible in bash either when you capture output with $(), unless you "tee", which opens up another can of "pipefail"-worms.- Working with data more than executing other binaries
- Dominant data processing path is structured (e.g. JSON, XML, binary)
- Script requires complex flow control to be manageable (e.g. functions, if, case)
- Shell UI requirements that involve handling command-line options
- You find yourself going _deep_ into the docs for jq to get the job done
.
Conversely, BASH is better under the following circumstances:
- Orchestrating other binaries the majority of the time
- Simple jobs that work with newline-separated text lines, or all data can be comfortably "stringly typed"
- Launching and job control for other processes
- You just need to glue some other shell commands and binaries together
- Are just manipulating shell environment vars, or providing a custom user shell (e.g. Python venv `activate`)
- Need to wrap a CLI tool in a simple way
.
The minute you have to reach for BASH arrays, case statements, math... stop everything and seriously consider using a language that has stronger support for all that.
I think the lack of batteries-included stdlib is probably the biggest holdup, but JS can be much nicer than python for anything that can be decomposed into map/reduce/filter string munging.
yes. I think if you find yourself dropping to sed/grep/awk/find too much, switch to python.
The language itself imho seems more readable. Strings, lists and hashes seem better especially when you choke on quoting.
And the python standard library makes a huge difference when you grow above "simple". I love argparse for readable arguments with help. os.path lets you manipulate paths and filenames in a readable and robust way. you can read and write json, csv and more. datetime lets you manipulate dates and times.
buggy_command || :
and 15 years later i am still begging people not to litter our deployment scripts with this pattern. i call this "the penny in the fusebox"
Nuke it from orbit, or something like that.
No one is prevented from using whatever shell they prefer for whatever purpose. For example I prefer to use a smaller, slightly modified scripting shell for interactive use. Some folks prefer to use a large, interactive shell for scripting.
As for the preferences of others, I note the default scripting shell on the largest Linux distributions is not Bash. It is of course Dash, which is derived from NetBSD sh, which is derived from the Almquist shell (ash). Indeed, ash is a popular choice for scripting. It is smaller and faster than Bash.
Edit: I take it back, the PDF metadata says:
Creator: PowerPoint
Producer: Mac OS X 10.4.8 Quartz PDFContext