Things I Wish I'd Known About Bash
zwischenzugs.com
zwischenzugs.com
My favourite little-known readline command is operate-and-get-next:
https://www.gnu.org/software/bash/manual/html_node/Miscellan...
You can use it to search back in history with C-r and then execute that command with C-o and keep pressing C-o to execute the commands that followed that one in history. Very helpful for executing a whole block of history.
For some reason, this documentation is hard to find! It's not here, for example:
http://readline.kablamo.org/emacs.html
I'm a bit saddened when readline replacements don't implement C-o. For example, the Python REPLs don't have it.
Didn't know about ctrl-o though, it sounds great! I hope that my ctrl-r override doesn't somehow break it.
[0]: https://github.com/junegunn/fzf
E: Fixed link.
I knew a quite a few commands but I didn’t know the block indentation commands and that was about a third of my typing over that week.
After that, part of my regular “I’m tired or there’s nothing to work on” routine included reading the accelerator key documentation for my editor of choice. I found all sorts of good stuff and it really made me faster for quite some time. Now I do more analysis work and the accelerators that help there are less numerous. Fast code nav being a critical one.
But some of these tools make it pretty hard to find all their features, which is a shame.
# a basic macro that types "foo" bound to META+"f"
"\ef": "foo"
# <META>-k
"\ek": shell-kill-word
# <CONTROL>-y
"\C-y": yank
# <SHIFT>-RIGHT
"\e[c": shell-forward-word
# <SHIFT>-LEFT
"\e[d": shell-backward-word
# <SUPER>-LEFT
"\e[D": "\e[d\ek\e[d^Y"
# <SUPER>-RIGHT
"\e[C": "\e[d\ek\e[c^Y"
# ^ ^ ^ ^
# | | | > yank
# | | > shell-forward-word
# | > shell-kill-word
# > shell-backward-word
ctrl-r, find a command, press ctrl-o -> inserts ^o in the command and exits of ctrl-r.
Running bash 4.4.12(1) on Linux.
At the least, rather than Bash, you might consider Perl as a default, lowest common denominator for scripts that need to run anywhere.
- It's nearly as ubiquitous as bash.
- It has approximately the same kinds of file/path operations built in.
- It has reasonably good support for strings/regexes/etc. all built-in, so you don't have to call out to tools like sed/awk/grep all the time and hope that they are available and compatible across your target platforms.
- It provides reasonably good arrays and hashes, which are horribly horrible in bash.[1]
- You can use syscalls very easily if you really need to, but usually you don't.
[1] Of course, no language can save you from the file system disaster (https://www.dwheeler.com/essays/fixing-unix-linux-filenames....), but being able to know that "foo bar" is a string instead of two array elements is a good start.
Mostly this all applies to Ruby or Python too, modulo perhaps the degree of ubiquity.
Perl regexes are the best of breed that everyone else replicates — far better than “reasonably good.”
The Perl erasure in this HN thread is startling.
"Erasure" to me implies some active effort to remove Perl from discourse. I don't see anything like that here: indeed, there are a number of positive mentions, and no negative ones I see. Granted, Python and Ruby are both mentioned more often, but none of those is at Perl's expense. Am I misunderstanding what you mean by 'erasure'?
I suppose being ignored beats the typical herp-derp anti-Perl bigotry, but I’d prefer all-around civility.
I'm having a hard time following you here. Do you think that they're doing so for any other reason that Perl is no longer their go-to tool? There are communities where Perl is still used: PostgreSQL for example uses Perl for some of its scripting, as well as its build farm tool, in particular because of its portability on older systems.
That said, from what I've seen over the past 10 years or so, Perl hasn't had much of a presence in areas where a lot of computer work in tech is being done. For example, in cloud computing, or scientific computing, or machine learning, or web frameworks. Please don't read this to mean that Perl couldn't be or isn't being used in these cases or wouldn't be a better fit. (As an aside, I think Perl missed out a lot while a large portion of the community was focused on Perl 6: there's only so much energy in a community, and that absorbed on Perl 6 wasn't focusing on evangelism. But that's not something I'm interested in litigating here.) Or that there isn't something a bit frustrating in seeing the wheel reinvented time and time again. And so many examples on the web use bash as a common denominator. This puts Perl further out of mind if it's not already part of your everyday workflow. And how many developers today have come of age without seeing Perl in their everyday environments?
Consider the current forum. What's the percentage of front-page posts that are about Perl or tools where Perl is a part of the tool chain? It would be understandable for the people who frequent HN to not view Perl as their go-to. I don't consider it uncivil for people to neglect to mention some other language when it's not something they'd actually think of reaching for. It seems the solution would be to share examples of where Perl provides advantages, both in the comments here and in submissions to HN.
If i had to choose only one of ruby/python or perl i would choose the former and it would be able to cover my base both as glue-code and for more programs. Perl would maybe make the glue code a bit easier but instead i would be much less employable and have a much harder time finding other people who can read the glue. I'm not qualified to have an opinion on perls capabilities for other programs but i'm sure there are valid reasons most people prefer other alternatives.
After using the linux command line (or its many «relatives» like cygwin, mac os, unixes) for more than 10 years now, I’ve talked to exactly one person that used perl to accomplish anything at all. He used it to edit text files, so he could have done the same in awk/sed/vim in my opinion.
I know more people who write fortran 77 than perl.
People just don’t seem to use (or like) Perl very much.
We support a lot of different OSes, and there's usually less variance between the Perl deployed on them than there is in the shell.
The problem is that Bash has about the lowest barrier to entry which can be found: just dump the things you were going to type anyway into a text file and mark it executable. It's simple, and then you bloody your nose on one of Bash's many idiosyncrasies, and people will say, "Oh yes, ']' is just a required last argument to '['. You gotta watch for that." And the true Bash master knows that my rule is silly and that anything may be written in Bash.
Just...please don't.
Do I need to manipulate rich data structures like hash-maps or nested lists? That sort of thing tends to stretch the capabilities of Bash to its limits and I tend to set the bar fairly low here.
Is the program oriented around commands? If I'm gluing executable scripts and binaries, using bash is often superior to a scripting language. Argument passing is more natural and convenient and the built-in support for the standard i/o streams makes it easy for the different commands to pass data between them. The number of variables or lines of logic is usually less important than the number of different installed commands I need to combine. (Or the number of variations on a single command)
Do I need to modify the environment in a significant way? In a bash script it's trivial to source in an environment script, which can be done conditionally or even interactively. This is usually more tedious to do in scripting languages.
Yes, you will have a few extra lines but it will be vastly more readable and maintainable.
And yes, I know I will get the standard the person who wrote the script did a bad job but at some point it should be okay to blame the tools instead of the workman if workmen disproportionately create worse results with a set of tools.
For scripts written in Python, I'd use a similar argument and suggest every serious Python scripter to learn Python. As for Perl, or Ruby, or Julia, or anything really. It's just that learning bash from its man page is, IMO, much easier than learning any of those languages to the same degree.
Of course – and it goes without saying — that some things are just not suited to bash, and in those cases a suitable language/framework must be used and learned if not learned. As far as process calling, environment management, or stdio streams management are concerned, bash is better suited than (all the other languages I've tried) Python, Ruby, or Go.
As a test, run shellcheck on any random shell-script written by these Coworkers from Christmas Past (or your own past) and it will spew serious warnings on almost every single line, run an equivalent analyzer on an equivalently unserious python file and you might in bad cases get 2 or 3 minor warnings per 100 lines.
The comparison is a bit unfair because just by getting these compiler errors and exceptions from a real language you force yourself into a more serious mental mode of programming instead of happy scripting, but that is just another argument in favor for not using bash IMO.
1. People who don't want to consider themselves "serious bash scripts" shouldn't be writing non-trivial bash scripts unless they're okay with it turning out buggy. I agree this is a personal standards thing, and reality is often less simple and more lenient than that.
2. The "compiler errors and exceptions" that you speak of in regards to Python also have equivalents in Bash. Agreed, they're still optional and non-"serious bash scripters" don't often know of them. Which is why I make my coworkers use them when I review their code.
3. There are classes of bugs that would happen in Python (and Go, from my experience) that wouldn't happen in bash, simply because they are in areas where bash shines. At work, I've seen process management and stdio management bugs — some of which have bit us in the field — simply because the non-bash language (Go) has a weird affinity to its child processes, or it (both Python and Go) defaults to not wiring up the stdio of child processes. In the latter case, the proper thing for the developer to do was the same as with bash: read the documentation. Most of our process management code is now in bash (because it's simple and safe) and systemd (because it's thorough and absolute).
But they know how to accomplish this task interactively in the shell, so scripting what they're already doing (or already know how to do) seems like the natural next step. So you wind up with an imperative shell script that's basically a long, flat sequence of commands with some logic and variables sprinkled in haphazardly as they realized they needed it.
Due to the organic way these scripts often emerge, it's not like advocating Python is an easy sell. By the time they think to consider alternatives, the shell version already exists.
If it's just a bunch of cuts or tr, sure.
Would the following be annoying for you to deal with?
#!/bin/sh
sed 's/#.*//' \
| sed 's/:/#/g' \
| cat AMD64 - \
| ./qhasm-ops \
| ./qhasm-regs \
| ./qhasm-fp \
| ./qhasm-as \
| sed 's/%32/d/g' \
| sed 's/%raxd/%eax/g' \
| sed 's/%rbxd/%ebx/g' \
| sed 's/%rcxd/%ecx/g' \
| sed 's/%rdxd/%edx/g' \
| sed 's/%rsid/%esi/g' \
| sed 's/%rdid/%edi/g' \
| sed 's/%rbpd/%ebp/g'
source: http://cr.yp.to/qhasm/qhasm-20061116.tar.gz qhasm-20061116/qhasm-amd64
And I would make an argument though that even large shell scripts in bash have their place.
I often write scripts in either Node or Python, but only when I need things bash is bad about (any sort of proper data structure beyond strings or arrays).
But there are just so many things bash makes insanely easy, especially with operating on files and directories.
And functions that are used as completions or need access to aliases or functions in the current process are also better in bash.
I wish there were a scripting language like bash, but enhanced with at least some hash maps and proper array manipulation, and maybe some formal IPC to allow scripts to request info from the parent process.
As someone who has inherited thousand-line shell scripts, and had to debug many 3rd party scripts, I stand by my assertion.
> Yes, you will have a few extra lines but it will be vastly more readable and maintainable.
Readability is important but it's not the only aspect to maintainability, nor is maintainability to sole concern of a tool. A low bug rate helps maintainability and actually having the features you need, in an acceptable timeframe, is also important.
For example, the OP mentioned the 'set -e' option that causes the script to exit if any command returns a non-zero exit code. In Python, you'd either have to remember to check the return code for every subprocess or define a wrapper, which adds complexity, reducing readability and can lead to bugs and errors. Nor is Python always the best answer for readability anyway. In many cases, it's not like it's just a few lines you're saving. Here are some functions I've used when scripting in Python
import subprocess, shlex
def process_run(cmd_string, stdin=None):
return subprocess.Popen(shlex.split(cmd_string),
stdin=stdin,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
def process_results(process_object):
(stdout, stderr)=process_object.communicate()
return (process_object.returncode, stdout, stderr)
def process(cmd_string, stdin=None):
return process_results(process_run(cmd_string, stdin=stdin))
f=SpooledTemporaryFile()
f.write(stdin_string)
f.seek(0)
results=process(cmd_string, stdin=f)
f.close()
return results
Actually what I'd say first is that it's quite possible the person writing the script knew what they were doing. I've inherited bad code in my life, I've inherited some real gems, and I've inherited a lot of code in between. One thing I've learned is that I tend to be unfairly critical of average code. It's hard to read unfamiliar code and easy to criticize inconvenient design choices when you have to adapt their code to some new problem that they never anticipated. Usually I'll be better off just buckling down and untangling the spaghetti.
Also, who is to say what I should be using "99% of the time?" Each problem has different constraints and different priorities.
Is there a reason you cannot just say `#!/usr/bin/env python3` in your scripts? I don't see why you require python3 to be the default, am I missing something here?
Again, in the context of this discussion, the whole argument is yet another point in shell's favor. There's no major backwards-incompatible change in the language. With Bash, you just decide whether POSIX compliance is something you need, and that's basically it. Both versions are still supported and no one interrupts discussions to announce that beatings will continue until morale improves whenever the deprecated version of Python comes up.
Or you could put the wrapper functions in a module and call it a day, either way problem solved.
> Do I need to manipulate rich data structures like hash-maps or nested lists? That sort of thing tends to stretch the capabilities of Bash
> If I'm gluing executable scripts and binaries, using bash is often superior to a scripting language
The two points above resonate with my view that there is a missing piece. On one hand we have bash which is optimized for being the glue (second point). On the other hand we have Ruby, Python, Perl, Go, etc which are good for first point. What I think is missing is newer, more powerful shell, which supports both use cases and more. I'm working on it:
Please note that I'm not the only one that thinks there is room for more powerful shells. See the readme for links to other projects.
Bash scripts are slow as hell (as most commands have to spawn new processes), it is hard to write "secure" code (if even possible), handling whitespaces can be a pain in the * and the amount of repetition is awful. If your kid has done something wrong, just tell it to write a bash script: it is the equivalent of writing a hundred times:
x="$(...)"
It is the glue which keeps everything working together.
Disclaimer: Please don't build a complete cathedral out of glue.
--JepZ
Doing things properly in Python or Go may take a few more lines (not much more really) but it is 100 tones more robust, and you need that if you are writing anything more than a 10-line one-off hack.
Exit codes are a poor substitute for proper error handling with verbose error reports. They do the job most of the time, as long as you remember exactly which command behaves which way. And that's a clear path to mistakes :-(
I think it's a fine guide. If I'm writing something that starts approaching a program rather than a few lines of utility-throw-away, it's time to at least immediately start co-developing in something more sane. My exception is for Makefiles, if one considers them "shell".
You pretty much have to put all the init stuff in a bash script. I can't think of any real-world examples of userdata scripts being less than 10 lines.
The alternative of course would be a three line script that downloads a file and executes it. But what language would that file be in? Probably bash. And it would make the infrastructure-as-code tracking and deployment process much more complicated.
You can use Go/Rust for static linked bootstrap instead of Bash. (See rustup.)
I'm thinking about combining GitLab (private token access + repository files: https://docs.gitlab.com/ce/api/repository_files.html#reposit... ) and a bootstrap script in Bash, that then launches something better. Plus self registration back into a GitLab repo (hey, it has the access token already, so it can push).
Yes, Ansible/SaltStack/Chef/Puppet reinvented, but you're now not bound to the horrible database, language, bootstrap, speed (slowness) and workflow of any of them.
If you want more access control, then creating a HTTP microservice that has the real private token (or proper OAuth2 access) to GitLab plus handles the self-registration and token handout/revocation is easy compared to the other parts. (And it can also store everything in GitLab.)
However, note that userdata will only run once at launch time. If you ever need to re-run userdata, you have to stop the instance, remove the userdata from settings, and then add again. YMMV.
I do the Python way when launching a beanstalk instance (basically cloud init).
I certainly prefer Perl for large scripts (Python is good too but I happen to know Perl better), but I often have to write scripts for targets that have bash but don't have Perl or Python. Since I've been forced to use bash, I've found it to be a better scripting language than I expected it to be.
parameter result
----------- ------------------------------
$name polish.ostrich.racing.champion
${name#*.} ostrich.racing.champion
${name##*.} champion
${name%%.*} polish
${name%.*} polish.ostrich.racing $ name="polish.ostrich.racing.champion"
$ temp="${name#*.}"
$ echo "${temp#*.}"
racing.champion
$ echo "${name#*ostrich.}"
racing.champion
$ awk 'BEGIN {FS=OFS="."} {print $(NF-1), $NF}' <<< "${name}"
racing.champion ${name%.*.*} polish.ostrich
${name#*.*.} racing.championI suggest reading this instead: http://www.grymoire.com/Unix/Sh.html
Granted, it's about the Bourne shell, but since Bash, Korn, and every standard UNIX shell is supposed to be compatible with it, it's well worth learning.
And IMO, if you need more than what the Bourne shell provides, you should be using a proper programming language like Python instead.
But, in my opinion, that's where it should stop: one shouldn't use a shell language for scripting. In scripts, it is simpler to use more verbose and clear constructs, because most editors are very powerful and provide shortcuts themselves.
>man bash
/\$\( # search pattern needs escaping
...
value is evaluated as an arithmetic expression even if the $((...)) expansion is not used (see Arithmetic Expansion below). Word split‐
...
n # go to next match
Command Substitution
Command substitution allows the output of a command to replace the command name. There are two forms:
$(command)
or
`command`
(detailed description follows)
To use the original example, once you've identified that $(...) is Command Substitution, you have something that pulls up meaningful results in every search engine.
type man foo
type /-p
type n n n n n n n n n
A presentation of man pages that used hypertext would make me a lot happier.
/ -p man man
man less
What helps me somehow is the fact that definitions in man pages are usually start on a new line and are indented by several spaces.
man bash
/ -o
man bash
/^ +-o
info is a greatly underused system and I'd recommend any *nix users to spend some time learning how to navigate it.
Kind of a deep rabbit hole, isn't it?
[1] https://www.gnu.org/software/bash/manual/html_node/index.htm...
From some other comments, it sounds like "info" uses emacs at its core? So I suppose I'd have to learn some emacs commands, if that's the case.
I'm surprised indexes aren't working for you - unfortunately I don't know of any suggestions to fix that.
This thread is a good reminder to give it another go.
man -k <keyword>It has been my experience that the definition of switches, unlike their use in examples or other text, occurs as the first thing on the line, so my search expression would be:
/^ *-p
It's possible the text will be tab-indented, in which case:
/^[ ^I]*-p[ ^I]
$ apropos timezone
Date::Manip::DM5abbrevs (3pm) - A list of all timezone abbreviations
dm_zdump (1p) - timezone dumper
Time::Zone (3pm) - - miscellaneous timezone manipulations routines
timezone (3) - initialize time conversion information
tzfile (5) - timezone information
tzselect (1) - view timezones
tzselect (8) - select a timezone
zdump (8) - timezone dumper
zic (8) - timezone compiler
-p
Potrzebie mode ...
/^ *-p
The surrounding context may be different, so adapt your search pattern accordingly.
If you’re using mandoc (http://mandoc.bsd.lv/) as your man(1) program—the default on OpenBSD and a couple of Linuxes like Void and Alpine—it will use these semantics to generate hyperlinks in the terminal using more(1) and less(1)’s ctags support.
So on my machine, your example becomes:
type man foo
type :t
type p
It translates to HTML too—check out the links generated by the web viewer, which uses the same backend. https://man.openbsd.org/ls.1
Amusingly, reference documents are themselves increasingly rare, and what you usually get is a rough tutorial and set of examples that cover perhaps 15% of the feature set, and need to dive into the source to figure everything else out.
Not having to put every command in quotes, for example, is a huge advantage all by itself, especially if any argument lists have quoted arguments.
The $(...) construct is pretty fundamental to bash scripting and should be covered in any decent tutorial.
COUNT=0
someCommand | while read -r LINE
do
COUNT=$(( COUNT + 1 ))
done
echo "$COUNT"
COUNT=0
while read -r LINE
do
COUNT=$(( COUNT + 1 ))
done < <(someCommand)
echo "$COUNT"
set -e
FOO=$(someCommand)
BAR="pre $FOO post"
set -e
BAR="pre $(someCommand) post" $ bash -ec 'BAR="pre $(someCommand) post"; echo hi'
bash: someCommand: command not found
$
Consider the following:
cd /tmp/
echo -e "hello world\nhello world\n:)" >> hello.txt
cat hello.txt
hello world
hello world
:)
bash -c 'sed s/"hello"/"hiiii"/ hello.txt | tee hello.txt'
cat hello.txt
hiiii world
hiiii world
:)
ssh localhost -t 'cd /tmp && sed s/"hello"/"hiiii"/ hello.txt | tee hello.txt'
cat hello.txt
Replacing the command with
ssh localhost -t 'cd /tmp && sed s/"hello"/"hiiii"/ hello.txt >> hello.txt'
hello world
hello world
:)
hiiii world
hiiii world
:)
What about using sed -i to make changes to the file, and not doing any bash redirection?
Alternatively, if the real problem is more complex than that, try using a different name for the output file and rename after it's finished.
Also, as mentioned in a sibling comment, your hello.txt is both an input and output.
is all you need
let count++
It catches so many errors that I think it's a must have in every programmer's toolbox. It even catches bash-isms when you are targeting POSIX sh. It saved me many many hours of grief trying to debug shell scripts I wrote and changed the way I write them for the better.
The documentation is especially great, for ever problem it detects you get an unique reference which you can lookup on the wiki eg: https://github.com/koalaman/shellcheck/wiki/SC2086 It then not only describes the problem but also shows different ways of solving it with some great examples and reasoning. I think I learned more Bash from Shellcheck than anywhere else.
I'm hard pressed to think of a modern unix that doesn't include bash by default. Solaris maybe?
Portability is only desirable, if you need portability - otherwise it frequently adds complexity for little return benefit.
`` vs $()
Another recent example I encountered is from the online book http://neuralnetworksanddeeplearning.com, there are some confusing ambiguities and plain misleading contradictions in NN terminologies, one being: multi-layer perceptions do not use perception neurons, these things can really screw with you while you're learning, especially the more subtle ones.
Some authors seem to have the ability to have full empathy for the novice while retaining expert knowledge and deep understanding by being able to both predict and answer relevant questions at the right point in an explanation at the right level of detail.
rename -n 's/(.*)/new$1$2/' *
There are two incompatible versions of rename in the wild:
1) Perl one: https://metacpan.org/pod/distribution/File-Rename/rename.PL
2) from util linux: http://man7.org/linux/man-pages/man1/rename.1.html
Debian (and dertivaties) ship the former; other Linux distros likely the latter.
The shell of course does expand unquoted globs.
1) This is a huge pet peeve of mine, but it kills me when my coworkers "bash" emacs and sing the praise of vim, then proceed to explain to me that ctrl-R in bash searches for the last command with a given pattern. They also often refuse to believe me that they're basically using emacs controls (because you know emacs' dirty). So, please, if you're in love with vim and use ksh or bash, learn about "set -o vi". Oh, and stop preaching!
2) "help xxx" for any xxx bash functionality, right there from the command line!
But not well implemented! You're supposed to be able to edit your search string, and I've never found a way.
Demo GIF here: https://unix.stackexchange.com/a/375914
* mac OS uses an ancient version of bash
* Embedded systems use busybox sh
* Android uses its own version of sh
* None of the BSDs come with bash, since, y'know GNU licensing
EDIT: Formatting
This comes from more armature script authors not being precise enough with requirements, or not targeting the more common and portable spec.
The shebang line #!/bin/sh should only be used for POSIX-compliant scripts; if it needs {bash, dash, python, etc.} it should start #!/usr/bin/env {bash, dash, python, etc.}.
But it's silly to think that there's nothing in between "complete control over target environment" and "use ANSI C so it can be compiled to any architecture and platform under the Sun." POSIX compliance will cover a lot. Try browsing /usr/bin on any Unix system you'll see plenty of use cases. I just looked at /usr/bin on my laptop and saw that mysqld_safe is a Bourne shell script, for just one example.
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...
On top of that you can use `checkbashisms` tool to lint your script against bash specific syntax.
And the accompanying reference site: http://shellhaters.org/
IMHO, we should use proper tool for the job instead of making artificial limitations. Bash is proper tool in lot of cases, unless old proprietary OS or very limited embedded OS are targeted.
That's not what I said
>we should use proper tool for the job
This is the implication of what I said.
I do not think that bash is the proper tool in most cases.
Chill. You don't have to hold bash so close to your heart.
i.e. all the *BSDs? Many Linux distros that ship with BusyBox only or FreeBSD/Linux type systems (AFAIK Debian and also maybe Gentoo allowed FreeBSD userlands)?
POSIX Sh is quite capable. I maintain even my bashrc POSIX compatible, in order to not have any problems when I want to run my config on some remote machine, or on some local BSD installation (and I did run FreeBSD for more than a year, quite feasible option for a workstation IMO).
There's nothing wrong with writing bash scripts, with a shebang like `#!/usr/bin/env bash`, just like there's nothing wrong with writing python scripts (with `#!/usr/bin/env python`), or Haskell scripts (with `#!/usr/bin/env runhaskell`), or whatever other language you like/think is appropriate/etc.
It's true that bash is unavailable by default on (say) microcontrollers, but I don't see the relevance given that loads of common scripting languages aren't available by default on microcontrollers (Python, Ruby, JS, PHP, etc.).
PS: With this said, don't start your bash scripts with a `sh` path like `#!/bin/sh` or `#!/usr/bin/sh`. In fact, don't use a hard-coded path like `#!/bin/bash` or `#!/usr/bin/bash` either, always use `#!/usr/bin/env bash` unless you have a good reason not to. (Whilst `/usr/bin/env` is a hard-coded path, it can also be treated as a single special-case by systems which don't follow FHS, like NixOS, GuixSD and GoboLinux).
$ grep somestring file1 > /tmp/a
$ grep somestring file2 > /tmp/b
$ diff /tmp/a /tmp/b
$ diff <(grep somestring file1) <(grep somestring file2)This is still wrong if the command can output spaces or meta-characters. You should quote the left operand, and then you don't need to prepend x:
if [ "$(grep not_there /dev/null)" = '' ]
Autoconf output is expected to run on buggy shells with broken empty-string comparison. Your scripts probably aren't.
if [ "$(grep not_there /dev/null)" = '' ]* word splitting and pathname expansion don't happen in [[...]];
* "==" does pattern matching in [[...]], but it does string comparison in [...].
-1000 for the parts that are specific to Bash. Stuff like that has been huge pain in my ass over the years. Some clever programmer uses some bash-ism and the build breaks on some ancient hardware that doesn't have bash.
I realize my complaint sounds like Henry Spencer's Ten Commandments and perhaps it feels outdated but trust me, you don't want to wade into thousands of lines of shell to track down why something doesn't work on the stupid AIX box.
A POSIX (or older Bourne-descended) shell can be distinguished from ‘bash --posix’ in a script with a fragment like “date&>F”, because Bash authors either didn't realize or didn't care that the sequence ‘&>’ already had meaning.
According to POSIX (2016 edition)
date&>F
date &
(exec >F)
date >F 2>&1Shouldn't the problem be the "ancient hardware that doesn't have bash" itself?
(I should add that I don't know what version High Sierra ships with but Sierra seemed to ship with 3.2.5x-ish which was 9 years old at the time.)
You can assume everyone has a modern bash, and make it the end users problem if they don't, or you can write portable shell scripts and know it will work.
Honestly the things you can't do in posix shell compared to bash border on "use a fully featured language" anyway.
[0]: I know POSIX is supposed to be even more widely available but depending on what you're targeting then it may not be the best option [source: https://en.wikipedia.org/wiki/POSIX#POSIX-oriented_operating...].
Thats the blocker on macOS.
The joke is that your shell is actually more portable than your shell script :)
If you use the standard ones (and use the "posix mode" flags when available) you're mostly ok.
Also, a shell script can have logic to handle different tools available (either different flavours of the same tool or even different tools that do similar things).
If the basic syntax it uses (or the shebang) are bash specific then you need bash to run it.
22:06 ~ $ bash --version
GNU bash, version 4.4.12(1)-release (x86_64-apple-
darwin16.3.0)
A posix conptible shell script has no such limitation. Thats all im saying. Im not arguing the merits of which bash is better or what a project might need.
The manual for bash consists of its Unix-style "man" page, and is therefore more of a reference than an instruction manual. I suggest using the Advanced Bash-Scripting Guide (http://tldp.org/guides.html#abs).
> It shows which scripts bash decides to run from the top, based on decisions made about the context bash is running in (which decides the colour to follow).
> So if you are in a local (non-remote), non-login, interactive shell (eg when you run bash itself from the command line), you are on the ‘green’ line [...]
With a bit of Googling I believe I found the origin of that diagram:
https://blog.flowblok.id.au/2013-02/shell-startup-scripts.ht...
The author there explains how the colors work:
> Fortunately, I’ve read the man pages for you, and drawn a pretty diagram. To read it, pick your shell, whether it's a login shell, whether it's interactive, and follow the same colour through the diagram. When the arrows split out to multiple files, it means that the shell will try to read each one in turn (working left to right), and will use the first one it can read
The remote bash startup order is further complicated by the existence of a compile time flag SSH_SOURCE_BASHRC. This flag determines if a remote non-interactive shell will load the ~/.bashrc file.
This flag is turned off by default and stays off in some distributions (like Archlinux), but is turned on in others (Debian, Fedora, ...) to replicate very old rsh behaviour.
echo '*'
echo "*"
if [ x$(grep not_there /dev/null) = 'x' ]
Press ESC then full stop instead. Less key presses.
Went searching how to do it in Zsh as well when the Zshell Line Editor (zle) is configured in vi mode:
$ bindkey -M viins '\e.' insert-last-word
Will make ESC-. work from insert mode.
. <(curl https://example.com/trusted.sh)
Background: A few years ago I was writing a Bash based OS installer. So after booting from a live CD I had to fetch the installer and execute it, which lead me to using that pattern frequently.
While I love it, I can't stress enough how dangerous it is.
make run-asan-test | c++filt
And even if the tests failed, the script would succeed.
The graph included seems to originate from https://blog.flowblok.id.au/2013-02/shell-startup-scripts.ht...
Instead of typing !$ for the previous command's final argument, you can use the keyboard shortcut alt+. (alt+period). Pressing it multiple times will go to the last argument of previous commands. I use this quite a bit and found it easier than !$, because you can see which command it will be : )
I still don't always understand exactly what is happening with subprocesses vs subshells (chriswarbo's post is very useful in pointing out how wrinkly this can be), so I try and keep bash my usage simple.
Blog post suggests he knew little of basic shell scripting until recently.
Blog also reveals he is selling a book on shell scripting with Bash, "Learn Bash the Hard Way."
I don't see how you could come to this conclusion based on the title. From the article, the author elaborates:
> "Recently I wanted to deepen my understanding of bash by researching as much of it as possible."
There seems to be a lot of good commentary in this thread indicating that people are finding the post useful. There are points here that I have. Do you have specific disagreements with the post contents?
`!*` and !gi:*If you declare a local variable and set it in the same step e.g: local MYVAR = $(/bin/false) The return code you get is from the local declaration, not assigning a value to the variable. It can be quite confusing when you afterwards check the return code with $? and it returns 0. Avoid it by assigning the value in a seperate command.
I have never used bash scripts professionally i.e. as a language rather than a simple script with just a command in.
Like wise back in 87 or so I got trained in sed and I have only used it once since, and that was when I was playing around with early linix's (when it came on a huge number of floppys)
I hadn't known that <(echo "hi") is treated as a file with the contents of stdout, which simplifies commands that take files as arguments.
This flag is turned off by default and stays off in some distributions (like Archlinux), but is turned on in others (Debian, Fedora, ...) to replicate very old rsh behaviour.
Regarding :h, is it any different than just using dirname?
dirname => "${name%/*}"
basename => "${name##*/}"
$ ls foo/bar/baz
ls: foo/bar/baz: No such file or directory
$ last=!$; echo "${last##*/}"
baz
$ echo "${last%/*}"
foo/bar $ echo foo bar
foo bar
$ echo !$
echo bar
bar
$ls /root/my-dir
[...]
$echo !:t
echo my-dir
my-dirhttp://vimdoc.sourceforge.net/htmldoc/cmdline.html#filename-...
$ echo foo /bar
foo /bar
$ !:h
echo foo
foo
This is what the author meant:
$ echo foo/bar
foo/bar
$ !:h
echo foo
fooInstead of
ls /long/path/to/some/file/or/other.txt:h ls !:$:h other.txt $ var=$(echo $'a\nb')
$ echo "$var"
a
b! was useful before command line editing; but not much since.
function() myfunc {
local foo="${1:-}"
if [ -z "$foo" ]; then
echo "Invalid first parameter!" >&2
return 127
fi
...
}
Instead, the quick and dirty way is to just "assert" via [parameter expansion](https://www.gnu.org/software/bash/manual/html_node/Shell-Par...):
function myfunc() {
local foo="${1:?First parameter must be provided}"
...
}
...and while we're on the subject of "set -e", I think that the ["unofficial Bash strict mode"](http://redsymbol.net/articles/unofficial-bash-strict-mode/) (putting "set -euo pipefail" and "IFS=$'\n\t'" at the top of your scripts) has been a bigger bug-prevention/rapid development aide to me than anything else. To be clear, I think it's a means of detecting some kinds of bugs. I've read Wooledge and others' objections to those patterns, especially "set -e", and agree with the point that this does not make your programs objectively safer and shouldn't be counted on as a crutch. Then again, neither does a linter, but it still helps you detect and avoid some kinds of bugs, so why not use it?
[ -z "$1" ] && echo "Invalid first parameter!" >&2 && exit 127
The most common bashisms and ways to avoid them are:
- Double brackets ([[) around conditions. Yes, I know that [ is a program (don't believe me? "which ["). That doesn't mean Bash uses it; it uses a builtin which is (almost) equivalent to [[ instead. Use that and your code will work in zsh/dash/all other POSIX shells. And while you're at it, stop using "which" as an authority for "is this a shell builtin or not?" [type()](http://linuxcommand.org/lc3_man_pages/typeh.html) is your friend.
- When comparing strings for equality, use a single equals sign "=", not "==" (e.g. 'if [ "$foo" = "some string" ]'). I know it feels dirty if you've programmed in any other language, but it changes nothing about your code's behavior and makes it compatible with several other shells.
- Don't use "function funcname()" syntax. It adds nothing over the basic "funcname()" syntax, but prevents your code running in many/most non-Bash shells. And consider putting your function-opening brace on a separate line (someone once told me that there are shells that won't accept any other function declaration style, but I've never seen one, so ymmv).
- Don't use "local" if you need to interoperate with ksh. Abandoning "local" pollutes global namespaces, though, so your call.
- Don't use substring expansion (e.g. extracting the 3rd-10th characters of a string via 'substr="${somevar:3:7}"'. That's not supported in many other shells. Alternatives include sed/awk/etc., or, if invoking external programs is absolutely unacceptable to you, something horrific like:
substr()
{
local input="${1:?String is required}"
local dist_from_start="${2:?Start position is required}"
local dist_from_end="${3:-${#input}}" # Here, it's actually 'offset', not distance.
local start_nulls=
local end_nulls=
dist_from_end=$(( 5 * (${#input} - ($dist_from_start + $dist_from_end)) ))
dist_from_start=$(( 5 * $dist_from_start ))
# Make a string of the regex for "any not null character" that "masks" the
# characters in the input before the start point, and the characters after
# the end of the substring. This is disgusting, and is only done because the
# parameter expansion statements can't contain repetitions (e.g. [^\0]{5})
# without the bash-only 'extglob' shell option.
# The not-null character is used because it will never match in a shell
# string.
while true; do
if [ "${#start_nulls}" -lt $dist_from_start ]; then
start_nulls="${start_nulls}[^\0]"
elif [ "${#end_nulls}" -lt $dist_from_end ]; then
end_nulls="${end_nulls}[^\0]"
else
break
fi
done
input="${input#$start_nulls}"
echo "${input%$end_nulls}"
}
substr "$@"
Anyway, some more bashisms: https://mywiki.wooledge.org/Bashism
- use the unofficial strict mode: http://redsymbol.net/articles/unofficial-bash-strict-mode/
- use parameter substitutions like ${foo#prefix}, ${foo%suffix} instead of invoking sed/awk
- process substitution instead of named pipes: <(), >()
- know the difference between an inline group {} and a subshell ()
- use printf "%q" when passing variables to another shell (e.g. assembling a command locally and executing it via SSH)
some-command <(some-failing-command)
Isn't that because the syntax provided doesn't use a pipe? Which is why actually using a pipe, rather than paren redirection, is easier on the eyes and the shell. It reads more like the flow of text:
maybe_failing() {
if ! some-failing-command; then
echo "That did not shake out" >&2
return 1
fi
}
maybe_failing | some-command
process-stuff --file <(make-input) \
--extra-data <(make-more-stuff | grep blah)
The point remains that <() was made for a purpose, yet lacks error handling, thus undermining its usefulness to the point of uselessness.
Edit: Isn't substitution using pipes, though? As in FIFO pipes? You get a file descriptor device which is closed at the end of the script. I don't know if the fd is wired directly to the command's stdout or whether the output is written in its entirety to a hidden tempfile first; the former sounds more natural and efficient.
Nope. <() and >() is equivalent to creating a named pipe with mkfifo and writing to it:
tmp_pipe_dir="$(mktemp -d)"
mkfifo "$tmp_pipe_dir/pipe"
make-input >"$tmp_pipe_dir/pipe" &
process-stuff --file "$tmp_pipe_dir/pipe"
But, the multiple case example highlights that if (for argument's sake) `make-input` and `make-more-stuff` had run _prior_ to that line, writing their (possibly empty) output to a (file|FIFO), how then would you want bash to behave? It would still open file descriptors to those (file|FIFO)s, which would still be just as blank.
It seems to me that if one wishes more fine grained control over the error handling in a multi-subshell-fd-trickery situation, then creating the FIFO(s) and managing the sender's exit status is the supervising script's responsibility.
a file descriptor device which is closed at the end of the script
I checked, and it's actually not even at the end of the script; those fds only exist for that one child, as `process-stuff` is exec-ed by bash.
whether the output is written in its entirety to a hidden tempfile
It's not a temp-file, it's an actual file descriptor, which bash `dup2`s for the subprocess, then cheekily uses the `/dev/fd/63` syntax to make appear as a file; you can peer into its brain a little:
$ showme() {
echo "showme.args=$@" >&2
the_fd="${1##/dev/fd/}"
cat <&${the_fd}
}
$ showme <(date -u)
showme.args=/dev/fd/63
Sat Jan 6 21:31:25 UTC 2018
Also: lol at HN downvoting advice to read a tool's docs. Bash is terrible for a lot of reasons, but not because of its lack of informative documentation.
Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
There seem to be things like "Testing" and "RTFM" that many people (including me) resist until they actually try it and see for themselves. I can remember the feeling of revelation when I finally learned to try "$ man foo" on everything...