Pure Bash Bible (2018)
github.com
github.com
Shell languages is the final layer of configuration and customization you apply to the setup, it should almost never be the foundation of what you build.
Especially for a company, unless you've got really, really skilled engineers on the scripts, it becomes a complete mess. And even then it can become almost impossible to grok.
The way we develop infrastructure automation, ci, and developer tools is binaries first and foremost, then plugins for our infrastructure (also binaries, or libraries) and then some zsh on top to provide a nice developer experience.
But we've had to heavily scale back our use of zsh, it simply turned into a ball of mud, like all software that isn't maintainable, upgrade-able, or simple enough to parse.
This isn't a dig on zsh, bash etc, we've just found out that we can't maintain zsh as a product to supply to our engineers, and as such have to correct.
¹ https://zsh.sourceforge.io/Doc/Release/Zsh-Modules.html#The-...
lines=( "${(@f)$(my_command)}" )
lines=( "${(@f)$(<my_file)}" )
But this is cool too.
> Especially for a company, unless you've got really, really skilled engineers on the scripts, it becomes a complete mess. And even then it can become almost impossible to grok.
Dude. It's really not that hard a language. Is amateur shell script ugly? verbose? redundant? bug-riddled? Absolutely. But it works despite that, and has fewer dependencies than other solutions, so in practice, it's fine.
That said, use whatever you know. Wanna use Python? Use it. Wanna use a compiled language? Do it. Just make something. The biggest difficulties you will have in producing a product will not be the language.
You don't have to avoid shell just because people on HN (who've probably never read the man page) keep repeating that it's terrible. With Shellcheck, almost anyone can write good scripts now. If you can write a concise, bug-free script that works, then do it.
No other method allows you to have a universal installer on every
platform that will always work.
Turns out on one non-Linux operating systems, one of the utilities used for string manipulation, while POSIX compliant, behaved differently than the same utility in nearly every other operating system.
has fewer dependencies than other solutions
Count how many programs are in /bin. Those are your dependencies. Unlike e.g. python or ruby dependencies, you'll have a much harder time upgrading /bin. Having to fork a new process for trivial tasks is also one of the reasons why shell scripts are so slow.
Sure, bash has builtin versions of some of those utilities like [, but not every system has bash.
It helps having good integration/e2e tests, but they only go so far, customers/developers customize and mess up their systems all the time.
Which is why I've choose binaries instead, they are mostly self-contained (except for various things about gnu and musl), and in general just rely on a higher level of built in functionality (because of the statically linked nature).
It is also quite bad that we've got to vet bash installation scripts before running them, as you may be required to run them as root. IMO, this isn't a huge issue for me personally, as we already are doing this for a lot of stuff, homebrew, linux package managers etc. We generally run a lot of untrusted code on our machines.
> Dude. It's really not that hard a language. Is amateur shell script ugly? verbose? redundant? bug-riddled? Absolutely. But it works despite that, and has fewer dependencies than other solutions, so in practice, it's fine.
Bash, zsh whatever, depend heavily on what is installed on the os, and what type of flavor of base tools are provided. Need base64 in a certain way, don't forget that line wrapping is different between bsd and linux. (a pain in the butt to test too).
Need to download something, better hope you're not running on a busybox or something with curl, or only have wget installed, now you gotta build support for both.
A lot of scripts, especially installation scripts are a product of blood, sweat and tears, mostly tears tho. Each line can be interpreted in a multitude of ways, under different conditions and versions of bash, os and so on, and the script will have to be robust enough for that.
> That said, use whatever you know. Wanna use Python? Use it. Wanna use a compiled language? Do it. Just make something. The biggest difficulties you will have in producing a product will not be the language.
It may not be the biggest difficulty, but it certainly is something we should consider. We've chosen our values, and shell scripts doesn't align with those, I'd rather not fight my environment every step of our development journey.
I am not building a new product, I am maintaining and expanding a product, as such I am heavily biased towards getting as few support tickets as humanly possible, while building an awesome experience. We are also mostly software engineers, and not sysadmins, as such we prefer compiled binaries. We then choose to distribute them through curated and secure channels.
We get the benefit that we can actually test parts of our tools in isolation, benchmark them if necessary, and version our stuff in a nice concise way. If I want to run our onboarding installation, I can simply fetch our current list of binaries expected for developers and run it, because I know they don't depend on external tools from the environment. This is a lot of work, but in a large development organization, this means a lot.
> You don't have to avoid shell just because people on HN (who've probably never read the man page) keep repeating that it's terrible. With Shellcheck, almost anyone can write good scripts now. If you can write a concise, bug-free script that works, then do it.
Just to reiterate, I am not digging on shell scripts, and I don't necessarily think they're terrible, they have their time and place, and for us, it is customization and extensibility. We setup our tools such that they work awesome on the cli, and can be composed however, developers like, but that is up to them, and how they want to form their workflows.
We've seen that developers really struggle to maintain what was previously built (myself included), so now we've got a more homogeneous tech stack, which comes with its own trade offs, but whose values align more with our engineering culture.
https://news.ycombinator.com/item?id=30363506 (291 points | Feb 16, 2022 | 96 comments)
https://news.ycombinator.com/item?id=21013150 (746 points | Sept 19, 2019 | 248 comments)
See also: Pure Sh Bible https://news.ycombinator.com/item?id=35765707 (406 points | 3 months ago | 133 comments)
If you have even the slightest hesitation regarding whether the task in front of you would be better done in a real programming language, the answer is do it in a real programming language.
set -e
Which forces the script to exit on first error.
This line be in the bible as Genesis 1:1
That should be line two of every bash script.
I like JavaScript. Because I'm good enough at it to ignore the bad parts. If I get good enough at bash, will it get better?
Bash itself will not get better because you'll be stuck writing scripts targeting some ancient version to ensure wider compatibility.
It's still a powerful skill to develop. It's a sweet spot between a one-off shell pipeline and something that involves actual logic.
Bash scripts should not be used for their logic. Think of them as a way to document a series of commands that you do not want to manually type. That can be including long argument names, generating includes or classpaths, or generating log file names based on the current time.
A good rule of thumb is if your script involves any if-statement other than parameter validation or basic on-success-of-x-do-y, it probably should be in a real language.
You might want to be in the dash POSIX shell for raw speed and portability.
1. Pipefail is a bashism, and doesn't exist on Ubuntu's /bin/sh or Busybox /bin/sh (common on embedded Linux systems)
2. Pipefail invalidates some common shell patterns like "grep _something_ | xargs -r CMD".
Pipefail is nice in some circumstances for sure. But it's probably not great advice to automatically copypasta into the top of every shell-script.
I wasn't suggesting for a generic posix shell. It's explicitly for bash.
If you're going to dumb down to pure posix, it's even more painful as you can't even have local function variables.
> 2. Pipefail invalidates some common shell patterns like "grep _something_ | xargs -r CMD".
That type of situation is the exception, not the rule. The more common situation is "foo some-input.txt | bar" and you want it to fail if foo fails.
> Pipefail is nice in some circumstances for sure. But it's probably not great advice to automatically copypasta into the top of every shell-script.
I'd much rather have that type of failure than an inadvertent continuation of a script when something early in a pipeline failed.
The defaults of bash make more sense for terminal use than scripting.
Powershell is a bit further towards the scripting end of the spectrum- part of why it annoys people coming from bash.
Compiled languages would be at the scripting end.
#!/bin/bash
set -e
grep nomatchforthis
echo "Won't get here!"function main {
if ! failing_cmd; then
log "this fails!"
return 1
fi
log "this won't log"
main
I only mention this because there are folks who don't like set -e.
https://github.com/zbm-dev/zfsbootmenu/releases/tag/v1.12.0
Don't fall in the trap of thinking things have to be written entirely in bash; it's okay to use other tools to help fill in the gaps.
EDIT: I didn't know of Bish, but from its code samples we seem to have converged on similar syntax :)
I do think it would be a really fun project. But is there really a practical use for it? Bash is portable but not that portable if you want to care about version compatibility. I understand that it's not always possible to install a different interpreted language like Perl or Python. Maybe the use case is for platforms where you can't install other language interpreters and can't cross-compile a compiled language? It seems like a very specific situation.
For anything bigger than a screenful of lines or something that would need proper unit testing or something that other people who are likely to be less versed in shell scripting need to maintain, think twice about using shell scripts.
Unixoid machines can use anything as scripts and using a language that breathes the programming language designs of the 1970s is not the best choice.
I find it useful when most of your work is wiring other command line tools together. When that's most of your work, shell scripting feels like it has the best affordances.
Anything that gets too fancy beyond this, with anything that resembles an algorithm or computation, you'll feel real pain :) Bash doesn't even support floating point!
A typical task in my day job is extracting metadata from and transcoding large AV archives, I always use a bash for this because I'm mainly stringing together tools like ffmpeg or sox, making directories, tempfiles, creating chesksum manifests, etc.
Also I typically do this on large CPU machines with many cores, I can parallelize operations in one line using xargs and a bash function. Doing stuff like this in python is actually a real pain in comparison.
There's a library for Node called Zx (https://github.com/google/zx) that provides a more shell-liked interface for Node.
I've been working on something in Python, inspired by Zx and Xonsh (https://xon.sh/), with syntax like this:
from pathlib import Path
from pyshell import Pipeline
args = ["123", "456"]
f = Path("out.txt")
with Pipeline() as p:
p.exec("a") | p.exec("b", *args) | p.exec("c", "x", "y") >> f
Is this something you'd use? It's slow going for me because it basically involves writing a compiler and I have approximately zero idea of what I'm doing. Knowing that other people would find it useful helps me stay motivated to keep working on it! Or if someone who theoretically is the primary audience doesn't find it useful, it might mean that I should spend my energy on other things.
Honestly I might be the wrong audience though, I've spent many years shell scripting and I find the flexibility of using the command line tools so great I choose to use it first despite the inadequacies of bash. I wonder if users with my point of view would ever be tempted away.
Also the shell is an interactive repl, which to me is where a lot of the power lies. The number of times I've asked a junior things like what the total duration of a set of audio files is, or the average length or something. They often come back with a python script or worse a jupyter notebook and pandas. The answer can also often be found by combining things like grep and awk in a few seconds once you've become acquainted with the tools, I guess I do try and demonstrate / preach this where I can.
But bc (and dc) does, and shell authors deemed this was good enough.
And that's part of the problem with the "Unix philosophy"
I had to do this once. It was so slow, I decided to use integers in kind of fixed-point math where I just multiply all the floats by 100 and work with ints.
zsh supports float though
If you prefer rigid adherence to standards, use awk.
I tend towards python these days if I need more than 1 variable since at that point I'm probably in the weeds already.
If it's just a ton of commands in succession though, I leave it bash. I have a few scripts in bash and PowerShell for weird ffmpeg operations I generally don't want to remember, stuff like loudness normalisation and converting to specific formats with specific parameters using ffmpeg.
If only to note, say, the section(s) of the Bash manpage or GNU Info Manual in which topics are discussed.
Taking the first-listed example, the '%%', '##', etc., substitutions are covered in Parameter Expansion.
I tend to go first to manpages, though I'm well aware that GNU deprecates these (exceedingly unwisely IMO) in favour of the Info Doc format (and its dedicated single-purpose reader). I'll note that the manpage alone renders to 120 pages (man bash | pr | less) onscreen, which is fairly formidable.
The PDF-formatted GNU bash manual (a fine format for a reference of this sort) runs 196 pages: <https://www.gnu.org/software/bash/manual/bash.pdf>
While there are exceptions, most packages have useful goodies in there to check first before one has to think about looking things up online.
What you'll find is both a set of pages & hierarchy of documentation as well as search (via swish++) through that space. And yes, there is an absolute trove of information available:
- Manpages
- Info pages
- Default package information
- Supplementary package documents, if installed
- HOWTOs from the Linux Documentation Project.
- Debian-specific documentation (installation, administration, security, policy, and other manuals)
- RFCs (divided into several groups, for manageability / relevance)
- Resources such as the Linux Gazette
There's also a commandline utility, dwww, which will invoke your browser of choice. Point that at a terminal-based browser (w3m, lynx, links, elinks, etc.) and you can quickly scan through docs from your terminal.
I remember, I used to keep a diary with landline phone numbers of relatives etc. I used to remember at least a dozen of those numbers for emergency. With mobile and digital contacts, now that number is down to just 2.
Wonder, what the programming is going to look like in 10 yrs. If no-one dedicates so much of their time and effort to document, the programmers then would feel helpless. right?
1. Or at that point, LLMs will be intelligent enough to read the source code and figure out the commands.
2. They can generate data by running commands, and train themselves using this simulated data.
What other scenarios will occur once we stop generating data using human effort because we all are so dependent on LLMs to generate those commands?
The solution is, do as little with shell scripts as possible to bootstrap a more workable environment.
Also you can use the chat as a learning tool, it's really a game changer for whom want to really learn it.
For learning in my case I needed to learn python and I'm already expert in javascript for example. So you can ask the prompt for the equivalent in python every time you feel stuck. But using the chat is so more precise and quick, than google for this kind of questions. Also without all the noise of google. In particular without this "noise" and time and concentration to find the response, it's a game changer when you really want to just learn and memorize.
Bash's dangers might be exacerbated by using ChatGPT, since you'll rely on the tool and you might not be able to understand what's wrong if shit hits the fan.
I'd recommend not using bash for anything other than the simplest tasks, as per Google's Shell style guide [0].
[0] https://google.github.io/styleguide/shellguide.html#s1.2-whe...
At this point just use something else.
I rather learn (or iterate) using a proper source of information (like a book, a blog post written by someone with actual knowledge, or even with a more experience coleague). At least for me it seems more productive.
Then try the same and find a human to explain it to you.
Then compare the price (including money and time) and performance of both.
I won't believe you made an objective assessment if your conclusion is that either party is the best solution all the time. The point is right now the number of occasions that AI has better price performance is large enough for it to be generally useful.
Why even ask then?
I have tried to use chatGPT a couple of time and many of those times it was wrong, to a point that if I ever use again I'd definitely double check the output(which makes me not bother using it in the first place).
Cute that some people find it useful, I don't, it has no use for me in the state it is today.
Than. Not “then,” than.
I point this out because I’m pretty sure lazy reliance on spellcheck and autocorrect are to blame with about half the internet using then when they should be using than.
I shudder to think what a few years of ChatGPT is going to do.
This page :
https://mywiki.wooledge.org/BashPitfalls
Is the fastest way of becoming proficient with shell scripting. Whether it's pure posix shell, or bash and its extensions, they're small languages and can be learned in a week end if you're proficient in another programming language. The main pain point is that there's a lot of "surprises" and non-intuitive behaviors, that are all listed here. The first ones, about parameter expansions, quoting variables and leading dashes are the most common issue with newbies in shell scripting.
There's a lot of amazing tooling these days to make sure your shell scripts are fine too. First, shellcheck:
I systematically run this on my scripts before using them.
Then shfmt for nice, reliable autoformatting: https://github.com/mvdan/sh
And finally, checkbashisms if you intend on making pure posix scripts that are compatible with debian/ubuntu's dash. It is part of the debian's devscripts suite, but is often individually packaged in other distros.
> Also you can use the chat as a learning tool
Or you could learn from a guide written by people who have suffered decades of experience of the pitfalls of shell scripting and have shared their woes.
https://mywiki.wooledge.org/BashGuide
chatGPT is good at coming up with magical solutions, but teaching you about all the corner cases?
Imagine using this "bible" to write a "pure sh" script thinking "this will be highly portable", only to have it fail because the Linux scripting shell is not bash in POSIX compatibility mode, it's dash. It would fail on NetBSD, too, whose scripting shell is the version of ash from which dash is derived.
Why?
I have no problem with people using #!/bin/dash shebang for their scripts and using dash as their default shell but it shouldn't pass itself as something else. It is fine with bash because it is bourne shell compatible.
The same can be said about bash.
> It is fine with bash because it is bourne shell compatible.
I was running an HPC cluster when we upgraded Debian during the bash->dash change, and there were all sorts of problems: IIRC most folks simply changed their scripts from /bin/sh to /bin/bash.
Then it doesn't sound fit for scripting.
In fact, there used to be a section which listed workarounds[1] for bugs in dash, but they have since been fixed[2]. If you are still using an old version of dash, you may need to use them.
[1] https://github.com/dylanaraps/pure-sh-bible/commit/70f410ebb...
Perhaps dash now has these features but does NetBSD sh or FreeBSD sh have them.
What's the point of "pure sh" if it's restricted to specific versions of shells. Opinions may differ but I'd rather learn the "lowest common denominator". I use the same scripts on both Linux and BSD so I need portability. I do not use bash for scripting. It's larger and slower.
One source I consult is https://www.in-ulm.de/~mascheck/bourne/
It is sometimes submitted to HN, but not IMO enough
https://news.ycombinator.com/from?site=in-ulm.de
I'm using the latest version of dash. I just tested it and it does have these ternary operators. However, I'm not going to use them in scripts unless I know they'll work on older dash versions, OpenWRT sh, Android sh, NetBSD sh, FreeBSD sh, etc. The process of updating these shells is generally very slow. It can take years.
"Pure sh" implies there is some advantage over "bashisms". I thought maybe it was portability but perhaps it is something else.
I like to write posix sh scripts for the sake of portability and, funnily enough, future proofing as I don't like having to maintain stuff against changes that break compatibility, which is something bash does (archlinux is still on an older bash even though debian stable has the latest, because bash 5.2 broke some of archlinux's own scripts. This is why you should not write bash scripts.), but bash being slow isn't one of the good reasons. If your scripts do that much work that your shell's speed matters, you should reconsider writing shell scripts and start thinking about using something like perl, python or ruby. I'd suggest perl, if only because unlike the latter, perl doesn't constantly pull the rug under you and make you do busy work to make sure your scripts work on the latest runtimes.
https://www.ruby-lang.org/en/news/2019/12/12/separation-of-p... https://nuclearsquid.com/writings/ruby-1-9-what-s-new-what-s...
But the language itself isn't the worst offender, the whole ecosystem around it tends to not even make reasonable attempts at stable APIs. Which isn't helpful with the way dynamically typed languages typically lack in tooling to safely, properly deal with refactoring - there are exceptions like smalltalk I've heard, but overall the cavalier attitude toward delivering a constant treadmill isn't pleasant to me. Of course, some may argue that the reason Perl doesn't suffer as much from it is because it is "dead", but I'll take dead over busywork.
I must say, I don't like Java the language, but Java the platform almost feels like the promised land compared to this. I don't use it to develop myself, but when I ran into an old unmaintained jar and ran it on a current JVM and it just works.. it feels.. fantastic. Absolutely wonderful. The same sort of feeling I also have when I run old games on Wine and it just works. win32 is the stable API I always wished linux had for GUI apps but will never have because the culture is all about CADT.
> If your scripts do that much work that your shell's speed matters, you should reconsider writing shell scripts and start thinking about using something like perl, python or ruby.
Usually yes.
But there is a small subset of use cases which need a really portable solution. Once I would have recommended to just write it in perl, because it was on virtually all systems. But now perl is getting phased out on some OSs/distros, and with python you never know if you get 2 or 3. Or embedded systems which don't have python/perl in the first place.
My go to solution for slow scripts is usually to limit subshells/forks as much as possible, and/or run it with busybox with the "exec prefers applets" option
edit: it's actually the "run 'nofork' applets directly" option. Can give quite a speed boost compared to bash if you have to call "external" utilities like grep/head/etc.
My hats off to you, I do not think I have the fortitude to stomach.. this.
> run it with busybox with the "exec prefers applets" option
> edit: it's actually the "run 'nofork' applets directly" option. Can give quite a speed boost compared to bash if you have to call "external" utilities like grep/head/etc.
Nifty trick, I had no idea about that, thanks for sharing this.
Yes and yes. My FreeBSD machine has not even been updated in 5 years, if that helps.
> What's the point of "pure sh" if it's restricted to specific versions of shells.
The aforementioned features have been implemented for a very long time. The issues with old versions of dash I mentioned were straight-up crashes for very simple things, not some fancy new-fangled feature that was yet to be implemented. Even then, the bible specifically listed workarounds for them.
dylanaraps has quite a prolific collection of shell programs, and they run on a variety of operating systems. Most notably, pfetch[1] runs on Linux, Android, NetBSD, FreeBSD, OpenBSD, Minix, Haiku, macOS, Solaris and IRIX. I assure you that he is fully aware of the importance of compatibility.
His pure Bash bible even has very thorough warnings for Bash versions required, since macOS uses Bash 3.2 (released in 2006).
I was under the impression that bash's POSIX mode was 'leaky': that a lot of non-POSIX stuff/extensions are available even though it's called as /bin/sh.
I know this caused a lot of our users issues when we upgraded Debian/Ubuntu during its switchover to dash. We told folks you could either rewrite those parts of the code or call /bin/bash.
$ man bash | sed -n '/BUGS/,/^$/p'
BUGS
It's too big and too slow.
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
bash can do whatever it wants to do if it's called as "bash". But it should only do specific things when called as "sh". This is true of any shell:
> When interpreted with dash instead of bash, the same script will fail. This is because dash is much stricter than bash in following the sh standard. Since dash is designed as a minimal implementation of the sh standard, it has to be stricter. The double brackets [[ … ]] are a ‘bashism,’ or a feature only available in bash and other, later shells such as ksh and zsh.
> Even though zsh also interprets most of these bashisms, zsh in sh compatibility mode is also stricter than bash and will error.
* https://scriptingosx.com/2020/06/about-bash-zsh-sh-and-dash-...
* https://zsh.sourceforge.io/Doc/Release/Invocation.html#Compa...
POSIX.2: Shell and Utilities (IEEE Std 1003.2-1992) is a few decades old now: regardless of when bash development started, I would think that by the 2010s it would have managed to better compartmentalize compatibility and extensions.
If you alias p=printf in a #!/bin/sh script, it will work; it will fail in a #/bin/bash script, unless POSIX mode is forced.
Once you need to do fancy array manipulation, hash tables, data structures, etc you're very likely much better off with Python, Perl, Ruby, Lua, Tcl, JS, or literally anything else. Notably, one or both of Python and Perl are available out of the box on Debian (and derivatives), macOS, most (all?) BSDs, and so on. Targeting Python 3.6+ will probably cover 99%+ of the systems you can find in the wild (e.g. Ubuntu 18.04 and newer).
I have also had some minor success using Go for that purpose. The toolchain can trivially cross-compile executables for ca 40 different OS/arch combinations; you need to consider the os/arch if you're running something different than Linux/x86-64 but that will again likely put you in the 1%. I'm working on some tooling to make this approach more straightforward.
I also have a fallback shell one-liner, without any of the fanciness like displaying the current git branch:
https://github.com/rollcat/etc/tree/master/cmd/prompter#i-li...