What was the point of [ “x$var” = “xval” ]?
vidarholen.net
vidarholen.net
$ a=1; b=1; c='!'; d='!'
$ [ "$a" = "$b" -a "$c" = "$d" ]
-bash: [: too many arguments
$ [ '!' '(' "$c" = "$d" ')' ]
-bash: [: `)' expected, found !
$ echo "${BASH_VERSION}"
5.1.4(1)-release
There is never a reason to use them, though. They can be replaced by the shell's && and || operators, which nicely bypasses the parsing ambiguities at the same time.
Not sure I agree. The equivalent of more complicated expressions like (a AND b) OR (c AND d) with && and || isn't always so "nice" IMO. e.g., try it for this:
[ -f a -a -f b -o -f c -a -f d ]
{ test -f a && test -f b; } || { test -f c && test -f d; }
Yes, it's more verbose, but the original expression isn't particularly readable either, and if you'd want to evaluate something like (a OR b) AND (c OR d) you'd need to introduce parentheses anyway.
([ -f a ] && [ -f b ]) || ([ -f c ] && [ -f d ])
(test -f a && test -f b) || (test -f c && test -f d){[ -f a ] && [ -f b ] ;} || { [ -f c ] && [ -f d ] ;}
And yes I know it creates subshells, that was precisely what I was saying in my comment where I said the simplest syntax I could think of (i.e. that very code) creates subshells.
$ a=1; b=1; c='!'; d='!'
$ [[ $a = $b && $c = $d ]] && echo true
true
#!/usr/bin/env bash
I expect every script with `#!/bin/sh` as shebang to be POSIX shell only.
Until your users/colleagues source it into another shell, or copy-paste some part of it into their shell/Makefile/Bazel/other script... and then either you feel sorry for them, or you laugh at them and blame them for burning themselves and trashing their system. (Hopefully your choice didn't cause their files to get wiped. Unless you think they deserved it for daring to paste your code somewhere else, I guess.)
The world isn't perfect.
Or ES2020 code in a browser.
Or ASM z80 into an x86 program.
When you're copy-pasting, you should be aware of what you're copying, and where you're pasting it.
Surely, no domain in IT is idiot-proof.
No, it is highly exclusive to Bash actually. In Bash, even one-character mistakes have a high potential to cause catastrophic damage. Just splitting or globbing on the wrong character can wipe your filesystem. As well as issues like these, where your string being '!' suddenly causes the command to be interpreted entirely differently. And Bash has modes that keep going when there is an error so it's not like you'll even know you've encountered an error or get any kind of visible crash + stack trace necessarily. It just keeps trashing everything further and further. These are extremely wildly different from Python.
> Surely, no domain in IT is idiot-proof.
I would love to know what happens when you say this to a user who pastes your Bash in zsh. Do you tell them your code isn't idiot-proof? Is everyone who ends up running Bash in zsh for some reason or another an idiot?
EDIT in response to your EDIT:
My users get either:
- a tar.gz archive
- a binary
- a docker image
- an helm chart
- a POSIX shell script (very rare)
You want your software to be portable? You have to put in the work to do it.
If you expect your users to know better (assume they ARE idiots), again, you're asking for something wrong to happen.
Blame the disease, not the symptom.
...because you can't delete important files unless you're root, right? Subfolders in your home directory are totally OK being deleted? You store your documents so that only root can modify them?
And do you know set -e has a ton of its own pitfalls and doesn't save you from this? It's a half-broken mitigation. I don't have energy to keep arguing here, but I would suggest go learning about the pitfalls of Bash and set -e and such.
You should use more copy/paste. ;)
The intersection of people knowing the "source" command and not knowing what is a she-bang or the difference between sh and bash is quite small.
When I write a bash script it is for my use or in my production system. In either case I call bash explicitly (/usr/bin/env bash) so the script will work correctly or not at all.
Rarely do I have a project where there is a requirement to write a "run anywhere" script but if it were I would stick to posix.
To me this sounds like saying we shouldn't have powerful tools like chainsaws because every year some people without safety training use them wrong and hurt themselves.
I'm only saying you should avoid using modern things needlessly. This means, don't do it when the POSIX-compliant syntax also serves your needs. e.g., don't do [[ -f "$foo" ]] when you could just as easily do [ -f "$foo" ]. And if you do need to use Bash-specific features, make sure it's safe enough that foreseeable misuses will fail safely. Or when that's not possible, try to mitigate it somehow (maybe with a comment, or by putting in some extra dynamic checks, or other solutions you can think of).
And this goes for other things too, not just Bash. Probably outside programming too.
I agree with that idea.
However, I don't think it applies in this case. The point of using modern bash syntax is that it fixed the brittleness and pitfalls of posix sh. This is significant. This is what I need in production. This is why our style guides insist on modern bash syntax for bash scripting.
I'd say the rest of your suggestions apply to shell scripts no matter what syntax is used.
if [[ -n "$BASH" ]] ; then
echo $0: use zsh, not bash
exit 1
fi
[[ -n "$BASH" ]] && { echo $0: use zsh, not bash; exit 1 }
[[ -n "$BASH" ]] && { echo $0: use zsh, not bash; exit 1; }My rule of thumb is that if your script becomes too complicated that you find POSIX sh limiting, it's probably a good hint that you should rewrite it in a proper programming language. Bash scripting is an anti-pattern in my opinion, it's the worst of both worlds since it's neither portable nor that much better than POSIX.
Writing modern bash using the updated syntax, it is easy to make robust scripts and the syntax around things like quoting is simple and consistent.
It seems odd to me to suggest that one should either stick to the old brittle /bin/sh syntax or not use bash at all. It has it's place in the world and it should be done the modern way.
You'll find 10s of thousands of lines of bash in the kubernetes repo and all the files I checked are written using modern bash syntax. There must be a reason they choose bash for these parts.
Of course, in any case everyone should use the tools they like and find most suited to the task at hand.
Kubernetes is a Google creation. Google mandates bash for shell scripts: https://google.github.io/styleguide/shellguide.html#s1.1-whi...
Also all of these modern are basically fancy wrappers around bash scripts. Even dockerfiles is just a way to run some shell commands at image build time and other shell commands at launch time. Much less puppet or ansible or user-data.sh ....
Bash is ported to basically every system, it's a small binary (by today's grotesque standards), and it gives you some added functionality that would be otherwise annoying to re-implement in POSIX sh (or Bourne sh). But it's also far less complex than Python. In today's world, if someone has any shell, it's a safe bet that they either already have Bash, or they can install it just like they'd have to install Python.
It's less painful in the long term to use Bash rather than Python. Python is more time-consuming and costly. It's a larger space requirement, you have to manage more runtime dependencies and a virtual environment, debugging is worse, and there's more opportunity for bugs because it's more complex. Bash scripts also rarely get bugs down the line due to dependencies changing, whereas this happens in Python so frequently we have to pin versions in a virtualenv. When's the last time you pinned the version of a Bash script dependency?
Also if we're in that gray zone of bash vs Python, you can pretty much stick to the stdlib.
glad to know i'm such a hellion running such irresponsible OSes! down with the man!! I do what I want!
Or more likely, the world isn't as clear cut as you might think
A distribution that removes Python 2.7 once upstream security support has been cut. Since Python is frequently used on network-facing services, it is unconscionable to allow its use past January 2020. Heck, one might even have setuid programs in Python and you don't really want to risk an unfixed local privilege escalation either.
> AWS Linux 2
A butchered version of CentOS
> and macOS
Not a distribution, and Apple doesn't exactly have a good track record with keeping things up to date or secure.
> Or more likely, the world isn't as clear cut as you might think
Given that the vast majority of the Python ecosystem is no longer compatible with 2.7, there isn't even a technical reason for kicking around 2.7 anymore.
Python 2.7 is trivial to build and run if you really, really want or need it, but since all the major upstream distributions have removed it, it sends a clear signal that you must take the risks into serious consideration before doing it.
/**
* cat.ts
*/
for (let i = 0; i < Deno.args.length; i++) {
const filename = Deno.args[i];
const file = await Deno.open(filename);
await Deno.copy(file, Deno.stdout);
file.close();
}What if you want to use something like delegator[0] to make running shell commands less toilsome?
*clearly, i've never taken a python CS style class, but it this importing core functions is just strange to me
"os" is a second-party library. You didn't write it; it was included in the Python distribution for you.
What you're asking, I think is why the "os" functions are not, in Python terminology, "builtins." The builtins are functions that are available without any kind of import: like len(), max(), sorted(), etc.
https://docs.python.org/3/library/functions.html
Why do you have to "import os"? Python was designed that way, that's all. So were the majority of other languages that come with standard libraries. (JavaScript is one exception that comes to mind, but there aren't many such exceptions.)
The discussion above relates to "virtual environments", which are ways to manage third-party dependencies. My claim is that virtualenvs, while handy for general-purpose development, are basically pointless for replacing shell scripts. You don't need them -- the core language and its standard library are sufficient for most shell scripting purposes. I'm basing this opinion on my ~20 years of using Python for this kind of work.
for example: anything with pipes
however once you start having to run multiple background jobs with custom wait logic it probably becomes worth it
from sh import wc, ls
print(wc(ls("-1"), "-l"))fgrep -f keywords huge.log | cut -f 2- | sort | uniq -c | sort -rn | head
Where size of data >> RAM (but less than free space in /tmp). It of course possible, but I suspect would require more effort and would work slower than a shell line above.
As for brevity, yeah, this particular example would require 5 lines instead of 1.
1. Ubuntu/Debian /bin/sh is the Debian variant of the Almquist shell, known as dash. The only non-POSIX element it includes is the "local" keyword (afaik).
2. Busybox, with the Almquist shell. Busybox also has a "bash" which offers a few cosmetic improvements to ash (an alias of [[ to [ is one that I can see in the source). I write POSIX shell scripts on Windows with busybox quite often.
The only other shell that omits everything outside of the POSIX specification is mrsh. I don't think mrsh is widely-deployed in any major distribution.
(You also aren't going to have bash in AIX/HP-UX and maybe a Solaris base load, but as the parent article says, we aren't talking about dinosaur herders.)
#2 is still currently tricky, but Rob Landley (former Busybox maintainer) is working on a full bug-for-bug compatible Bash clone called toysh which will be included in an upcoming release of Toybox[1]. Once that’s released, I’m looking forward to (hopefully) never writing a script for BusyBox ash again.
For pure programming, “real” languages are absolulely preferable in almost every way, but all of them fail when it comes to running external programs and redirecting their output in a way that doesn’t make me pull my hair out. As a heavy user of both “real” programming languages and shell scripting languages, I’m left craving something that brings the best of both worlds.
There are a number of newer shell projects in this vein that are very exciting, like Oil [0], Elvish [1], and Xonsh [2]. I was also hopeful that Neugram [3] would go somewhere, but the project seems to have died out. While many people cite Bash’s lack of portability as a reason not to use it, I find Bash to be very portable for my use cases and avoid using these newer shells for their lack of portability. Maybe one day we can have nice things.
[0]: https://github.com/oilshell/oil
[1]: https://github.com/elves/elvish
Bash acts like 'historical versions of sh' if it was invoked as sh. So, a Bash replacement could act like Bash if it was invoked like bash. Then implement all the extra crazy shit if you invoke it as slash or something.
Assuming it was a small, fast, compiled binary, this would take off in all the distros pretty much immediately. And if you really want it to succeed, implement a spec first, and implementations second. Add a slash implementation to Busybox, and then it's on every embedded Linux system in the world.
Personally I think the better approach is to accept that bash/sh will always be about for legacy stuff and for alternative shells to carve out a niche elsewhere. Particularly because to some of bash/sh's pain points can't be addressed without break compatibility in the first place (like handling file names with spaces).
> Oil is a new Unix shell. It's our upgrade path from bash to a better language and runtime.
[4]: https://github.com/lmorg/murex
Superficially it's quite similar to elvish (purely by coincidence) but it's aimed around local machine use (eg by developers and devops engineers) so is as much inspired by IDEs as it is by shells.
Try https://amoffat.github.io/sh/
from sh.contrib import git
git.reset(hard=True)The doc says: sh is a full-fledged subprocess replacement
It would be nice if that page had some sort of comparison. I guess the examples shown are sufficient if you're a regular user of subprocess?
Development of the formal ksh93 was halted and rewound by AT&T after David Korn's departure, precisely because the user community remains extremely intolerant to changes in functionality and/or performance. Korn shell development that requires a "living" shell should likely target both ksh93 and the MirBSD Korn shell (the default shell in Android).
BASH has diverged from the Korn shell in a few (annoying) ways, but remains core to the GNU movement and rightly commands its own audience. It is much larger than the MirBSD Korn shell.
POSIX shells were developed to target truly minimal systems - POSIX rejected Korn likely due to (Microsoft) XENIX on 80286 systems that only allowed 64K text segments. Korn was able to run on such machines, but the code was not maintainable. Clean C source for a POSIX shell is far easier to achieve for XENIX, and remains a better fit for embedded environments. This is likely why POSIX is not Korn.
I don’t really care about targeting, say, IRIX, or z/OS, or the early-2000s-era Windows POSIX Subsystem. But I do care about targeting Linux, macOS/iOS, all the modern living BSDs, and Android (where applicable.) I also maybe care about letting my software run under a Linux Busybox userland.
It’d be great to know what the lowest-common-demoninator standard is for just that set of targets. I don’t care about POSIX, but if that set of targets had a common formal named standard, I’d adhere to it rigorously.
Since it doesn’t, I just mostly adhere to POSIX — except where it gets to be too much (like having to write 1000-line Bourne Shell scripts.) When that happens, I look around to see if the set of targets I care about all do something the same way. If they do (such as all happening to ship some — perhaps ancient — version of Bash), then I break away from POSIX and take advantage of that commonality.
I don’t know enough about all the things these OSes do or do not have in common to truly code to the implicit standard 100% of the time. I wish I did. So, in most respects, wherever I haven’t done independent research, I have to hold myself to a much stricter standard — POSIX. But it’s only ignorance keeping me doing it!
Sure, those are very niche, but not all BSDs or every Linux distros have bash in the base distribution. And if you work with embedded system bash is very much a luxury. You mention busybox but this one is very much a crap shot because everything in busybox is opt-in. You want to use `seq`? Better make sure that it's configured. On the other hand you almost certainly won't have bash on these systems either.
At any rate if the alternative is between having to install python, ruby, perl or lua on my machine or having to maintain and hack on ginormous bash scripts I know what I'll chose.
But in general I like your approach, it's pragmatic. It's just pretty hard these days to find a system that ships bash but can't give you some decent scripting support, at least in my experience.
docker run --rm -it alpine
/ # bash
/bin/sh: bash: not foundThe worst part of Linux development is the insistence on 'portability' when using the Unix tools. As a result bash, make, awk, have all stagnated.
You would probably find also python, but what version? Python3? These days you are probably guaranteed to find them, except that there are old Red Hat servers still in production where is not installed. And Python3, what version? 3.7? Older? What features are safe to use and what are not? Do you have to research to find out? Or you are stuck with using Python 3.2 just to be safe? Well, in the time you wasted thinking about that, you would have finished writing the script in bash.
Totally agree here. Shells are good for one thing: user interaction. For all the rest, use a programming language.
1. interactive file listing and tab completion
2. the ability to run one-off lines
3. the ability to run parts of scripts and build up into a bigger script
Currently I write a lot of small-ish scripts using either bash or bash plus a text editor. I would like to use a less obtuse language than bash, but the interactive part is really important. I don't want a system where I have to run exploratory and test commands in bash and then translate them into a different language, because that means figuring out how to say it twice and doing extra debugging.
Also bash is not portable, but you can assume to find a recent version of bash on any Linux system. And for BSD systems or macOS, you can install it with a command. And yes, there are other ancient UNIX systems, but who uses them? But what other programming language can you be sure to find on any system?
Finally, bash it's quite efficient. I mean that it doesn't have the weight that a full programming language like Python has, especially the time that it takes to activate the program interpreter. That could make a difference in scripts that are executed a lot of time from other script or other programs. Or even in wrapper scripts, since they doesn't add noticeable delay when executing a command.
There is still a reason to use and know bash to me. Of course implementing whole programs in bash doesn't make sense, but for simple scripts it does.
None.
from sh import sort, du
from glob import glob
print(sort(du(glob("*"), "-sb"), "-rn"))I think it's better said that anyone writing bash scripts today should always pass their script to shellcheck.net. It can even be installed locally in case sharing the script online is undesired.
Shellcheck will not only look for problems in the script but it will also usually provide some background about how the problem is a problem and a suggested workaround or solution.
Well, mostly:
$ a=1; b='*'; c='!'; d='*'
$ [[ $a = $b && $c = $d ]] && echo true
true
$ [[ $a = "$b" && $c = "$d" ]] && echo true
$
> When the == and != operators are used, the string to the right of the operator is considered a pattern and matched according to the rules described below under Pattern Matching, as if the extglob shell option were enabled.
POSIX syntax is more consistent and more portable:
$ [ "$a" = "$b" ] && [ "$c" = "$d" ] && echo true
$Depends on what you mean by quote safe. Quotes aren't redundant:
$ d='*'
$ [[ foo = $d ]]; echo $?
0
$ [[ foo = "$d" ]]; echo $?
1I would call it a bug when there's one valid way to read it, and [ fails to see it. I don't see the ambiguation. For example, zsh's [ doesn't have these errors:
$ a=1; b=1; c='!'; d='!'
$ [ "$a" = "$b" -a "$c" = "$d" ]; echo $?
0
$ [ '!' '(' "$c" = "$d" ')' ]; echo $?
1
I think bash's [ is just lacking the ability to backtrack while parsing its arguments.
Curiously, GNU's [ fails to parse the arguments of the first, but not the second's:
$ command [ "$a" = "$b" -a "$c" = "$d" ]; echo $?
[: extra argument ‘!’
2
$ command [ '!' '(' "$c" = "$d" ')' ]; echo $?
1
$ command [ "$a" = "$b" -a '(' "$c" = "$d" ')' ]; echo $?
0 [ x"$var" = x"val" ]
(It behaves the same way since the shell allows quoting to start in the middle.)
http://www.oilshell.org/blog/2017/08/31.html
The POSIX spec did indeed make things cleaner; the last section quotes it and gives some style advice.
Moreover, similar problems are inherited by some programs that try to do shell-like commands but put their own syntax in the way. E.g. Ansible, where YAML, JSON, module call DSL and Jinja are all mixed on top of the commands so you can bump into any of them.
Actually re-reading your post you probably know that already! Nevermind then. Maybe someone reading this will be enlightened
I imagine the introduction of `[[` and `((` in Bash may have come partly from this reasoning (to abandon the POSIX semantics for these commands), though I never delved far enough into nerdery to learn the difference from `[`—so don't know if they do evaluate vars in the sensible way.
I keep wondering if there are any shells or other software calling into utils, that manage to walk the edge of writing commands without bumping into such syntax problems—but avoid having to "enquote" "each" "argument" all the time.
At some point, "decades old idiomatic hack that increases robustness" stops being a hack and becomes an idiomatic way to increase robustness.
A better conclusion would be "yes, keep using it".
You know, shell scripts are a hack.
I really wish there was a middle ground between shell scripts and say python. Python has proper lists and hashes and loops and can manipulate paths with spaces or quotes or unicode. It doesn't get dragged down by the escape-an-escape-within-a-regex nonsense that makes things undecipherable. or ${VAR:-foo} or ${VAR##ugh}
But python is a bit fumbly when it comes to invoking external programs. try: ... subprocess.bleh except: ... ugh.
e.g. this post has the slogan that it's "for Python and JS programmers who avoid shell".
http://www.oilshell.org/blog/2020/01/simplest-explanation.ht...
Oil has hash tables, and can manipulate paths with spaces -- in fact the latest blog post is about that!
http://www.oilshell.org/blog/2021/04/simple-word-eval.html
Also the same blog post mentions that we should get rid of ${VAR##ugh}. Help wanted :)
I actually tend to prefer that these days — if I write shell I'm tossing in "set -e -u -o pipefail" anyway and subprocess's newer APIs end up being cleaner when you need to do anything non-trivial for error handling (I usually end up with one `try:` block for most of the program and maybe one or two `except: cleanup(); raise` blocks. Having the logic be clean and consistent saves you so much time when you revisit that program a month later.
The biggest wins I've seen from quoting external program arguments — either not needing to do so at all or passing things cleanly through shlex.quote makes it so much easier when you have the possibility of any special characters in arguments.
The exploit was based in inserting some \$\( make_a_new_root_account_in_etcpassw \) inside the variable inside the test.
As a result, both my friend and I stopped writing shell scripts as CGI and started learning perl just to do it.
Given i haven't spent a lot of time writing shell script yet but the syntax is just super weird, like I recently learned(1) that when you When you write if [ -e /etc/passwd ]; then .. that bracket is not shell syntax but just a regular command with a funny name.
Is there a recommended bash transpiler like what babel is to Js? So i can write bash scripts using a more coherent language and it spits out .sh files.
My "favorite" (i.e. most terrifying) BASH fact is this: https://unix.stackexchange.com/questions/121013/how-does-lin...
I do this because my shell scripts use bash specific features, and I want to distinguish them from portable shell scripts that use POSIX shell features only. The latter being the only files that really deserve being named as .sh in my eyes :)
That’s how it used to be. Nowadays, most of the time it is a built-in, not it‘s own executable anymore.
Perl may look ugly, but it doesn't have the escaping problems that Unix shells have. And most of the usual shell features (listing files, launching commands, etc...) are built-in. It is also available in almost all Unix systems, maybe even more so than bash.
That's a bit of a deceitful claim - maybe bash isn't on every UNIX box but /bin/sh sure is along with awk and cut and all the other associated helpers.
Even if perl is installed its likely missing all the cpan modules that make it useful which isn't great if the host doesn't have carte blanche access to install whatever it feels like.
As a sysadmin/devops person knowing the utilities that are guaranteed to be installed is essential, especially now that docker images trimmed of any fat are trendy for bogus security ideals.
That's what I'm using for github-keygen for maximum portability. https://github.com/dolmen/github-keygen/
Not suggesting it's suitable for everyone though. But it's certainly solved a lot of problems for me.
It is a bit sad that his problem has to be encountered and resolved over and over and over again. Granted, it was a SIMPLE PARSER because resources were extremely tight. Note, I don't mean shells, I mean working with projects that implement parsers. As soon as someone busts out lex/yacc I know I'm going to see at least one "ah, we didn't think of that" issue. Parsers are hard.
So when test sees that one of its arguments is the string "-f", it has no way to know whether that string came from a variable like $var, a quoted string literal like "-f", or just the text -f. Most of the time if you're writing a parser for your own DSL you don't have this problem. You can tell the difference between the string literal "+" and the operator +.
They can be written in any compiled language you like or even an interpreted language, if you are prepared to add the dependency of an interpreter for that language at boot time.
This is a lot simpler than migrating to (and getting locked in to) systemd.
Init scripts where simpler because the boot process was well defined, meaning that you had all your scripts in rc.d that were executed in that particular order.
Now with systemd is more difficult to have something start after something else, you have all the dependencies and if you get them wrong it will not work. Worse, the boot process is no longer deterministic, meaning that 99 times it could work and it could break the 100th time.
On the other side, systemd is a necessary evil for a modern desktop system, where you have multiple events that could fire (hotplug, network, power events, etc). But not so sure if it's that necessary for a server or an embedded device.
Not really, it's just one simple line in a unit file. (https://fedoramagazine.org/systemd-unit-dependencies-and-ord...)
Claiming "it is just 1 line" is either inexperience or dishonesty.
In the end, the init script style is just the bare minimum that works solely by POSIX features and is a PITA when it comes to advanced use cases like dependencies or machine-readable system state or platform independent init definitions. Systemd is certainly not the ultimate tool ever written, but it is a bold step in the right direction. Maybe someone will rewrite it someday (if so, presumably in rust) and I will appreciate the effort to make things even better.
Depends on your perspective - for a developer, “do whatever you want in whatever way you’re comfortable with” is simple. As a sysadmin, I much prefer the simplicity of “there is one way to do it; features like resource limits and automatic restarts work out of the box, and are consistent across all services”.
Oof. If people think JS is bad wait until they try to do anything moderately complex in shell script
The number of times I've started with a simple bash script and regretted it down the line. Now I refactor into perl far earlier in the process
For all the vaunted Unix Philosophy it's amazing how clunky some of it is. Countless shell scripts still trip over spaces in filenames. Of those that don't, nearly all of them will still be confused by any unusual characters like newlines. If you want something done properly, you have to pull out a proper scripting language, to have readdir() and the ability to pass arguments directly to a process.
Even Perl, which generally excels in such tasks has weird lapses in convenience. You can run a command with one line of code without the possibility of confusion with `system("ls", "-l", $dir)`, but you can't get its output that way. There's no version of `system` that'd allow you to both explicitly specify each argument to the process, and obtain its output. You either use backticks and risk quoting trouble, or need to use `open`, which is a lot more verbose.
It's interesting that it took Microsoft to do a new approach in this regard. PowerShell has its amount of weirdness I really hate, such as that the environment escapes the program into the commandline, but it's really refreshing how it dispenses with brittle grep, awk and cut stuff.
You can:
- specify each argument separately
- get separate stdout and stderr
- provide input
I'm just wondering at that bizarrely enough, Larry Wall (or somebody else) found it useful to have a simple, convenient way to execute a command with exact arguments, but not for when you need its output.
I must have written the same wrapper around open() several dozen times by now, due to needing to write scripts for cases where installing dependencies is undesirable.
It had backquotes from the start. So it was as convenient (and as unsafe) as the Bourne shell.
Of course you quickly want more safety and backquotes are just legacy you can never use for serious stuff.
I'm not saying it can't be done, just that the language is oddly lacking in convenience for this use case.
You loose.
https://en.wikipedia.org/wiki/The_UNIX-HATERS_Handbook
http://www.art.net/~hopkins/Don/unix-haters/BarfBag.gif
https://web.mit.edu/~simsong/www/ugh.pdf
Preface
“I liken starting one’s computing career with Unix, say as an undergraduate, to being born in East Africa. It is intolerably hot, your body is covered with lice and flies, you are malnourished and you suffer from numerous curable diseases. But, as far as young East Africans can tell, this is simply the natural condition and they live within it. By the time they find out differently, it is too late. They already think that the writing of shell scripts is a natural act.”
— Ken Pier, Xerox PARC
The Shell Game, p. 149
Shell crash
The following message was posted to an electronic bulletin board of a compiler class at Columbia University.
Subject: Relevant Unix bug
October 11, 1991
Fellow W4115x students—
While we’re on the subject of activation records,
argument passing, and calling conventions, did you
know that typing:
!xxx%s%s%s%s%s%s%s%s
to any C-shell will cause it to crash immediately?
Do you know why?
Questions to think about:
• What does the shell do when you type “!xxx”?
• What must it be doing with your input when you type
“!xxx%s%s%s%s%s%s%s%s” ?
• Why does this crash the shell?
• How could you (rather easily) rewrite the offending
part of the shell so as not to have this problem?
MOST IMPORTANTLY:
• Does it seem reasonable that you (yes, you!) can bring what
may be the Future Operating System of the World to its
knees in 21 keystrokes?
Perhaps this is why Unix shells don’t let you extend them by loading new object code into their memory images, or by making calls to object code in other programs. It would be just too dangerous. Make one false move and—bam—you’re logged out. Zero tolerance for programmer error.
The Metasyntactic Zoo
The C Shell’s metasyntactic operator zoo results in numerous quoting problems and general confusion. Metasyntactic operators transform a command before it is issued. We call the operators metasyntactic because they are not part of the syntax of a command, but operators on the command itself. Metasyntactic operators (sometimes called escape operators) are familiar to most programmers. For example, the backslash character (\) within strings in C is metasyntactic; it doesn’t represent itself, but some operation on the following characters. When you want a metasyntactic operator to stand for itself, you have to use a quoting mechanism that tells the system to interpret the operator as simple text. For example, returning to our C string example, to get the backslash character in a string, it is necessary to write \\.
Simple quoting barely works in the C Shell because no contract exists between the shell and the programs it invokes on the users’ behalf. For example, consider the simple command:
grep string filename:
Searching for strings that contain periods or any pattern that begins with a dash complicates matters. Be sure to quote your meta character properly. Unfortunately, as with pattern matching, numerous incompatible quoting conventions are in use throughout the operating system.
The C Shell’s metasyntatic zoo houses seven different families of metasyntatic operators. Because the zoo was populated over a period of time, and the cages are made of tin instead of steel, the inhabitants tend to stomp over each other. The seven different transformations on a shell command line are:
Aliasing alias and unalias
Command Output Substitution `
Filename Substitution *, ?, []
History Substitution !, ^
Variable Substitution. $, set, and unset
Process Substitutuion. %
Quoting ',"
Having seven different classes of metasyntactic characters wouldn’t be so bad if they followed a logical order of operations and if their substitution rules were uniformly applied. But they don’t, and they’re not.
[...followed by pages and pages of more examples like "today’s gripe: fg %3", "${1+“$@”} in /bin/sh family of shells shell scripts", "Why not “$*” etc.?", "The Shell Command “chdir” Doesn’t", "Shell Programming", "Shell Variables Won’t", "Error Codes and Error Checking", "Pipes", "| vs. <", "Find", "Q: what’s the opposite of ‘find?’ A: ‘lose.’"]
My judgment of Unix is my own. About six years ago (when I first got my workstation), I spent lots of time learning Unix. I got to be fairly good. Fortunately, most of that garbage has now faded from memory. However, since joining this discussion, a lot of Unix supporters have sent me examples of stuff to “prove” how powerful Unix is. These examples have certainly been enough to refresh my memory: they all do something trivial or useless, and they all do so in a very arcane manner.
One person who posted to the net said he had an “epiphany” from a shell script (which used four commands and a script that looked like line noise) which renamed all his '.pas' files so that they ended with “.p” instead. I reserve my religious ecstasy for something more than renaming files. And, indeed, that is my memory of Unix tools—you spend all your time learning to do complex and peculiar things that are, in the end, not really all that impressive. I decided I’d rather learn to get some real work done.
—Jim Giles, Los Alamos National Laboratory
$dir = '-r';
You loose.
Quoting is not enough. You also have to fully understand the syntax of each command.
Fix: system("ls", "-l", "--", $dir)
And while we're being pedantic, it's 'lose'.
Fair. I'm better at shell and Perl languages than English. None of them are my mother tongue.
https://www.bourguet.org/v2/pdp10/cmds/chap1
https://en.wikipedia.org/wiki/Command-line_completion#Histor...
https://en.wikipedia.org/wiki/TOPS-20
Command Processor. Rather advanced for its day were some TOPS-20-specific features:
Command completion. Dynamic help in the form of:
noise-words - typing DIR and then pressing the ESCape key resulted in DIRectory (of files)
typing I and pressing the Esc key resulted in Information (about)
One could then type ? to find out what operands were permitted/required.
PCL language features. PCL includes:
flow control: DO While/Until, CASE/SELECT, IF-THEN-ELSE, GOTO
character string operations (length, substring, concatenation)
access to system information (date/time, file attributes, device characteristics)
https://web.archive.org/web/20200115181431/http://www.opost....
A Brief Description of Recognition and Help
Typing the escape key says to the system, "if you know what I mean from what I've typed up to this point, type whatever comes next just as if I had typed it". What is displayed on the screen or typescript looks just as if the user typed it, but of course, the system types it much faster. For example, if the user types DIR and escape, the system will continue the line to make it read DIRECTORY.
TOPS-20 also accepts just the abbreviation DIR (without escape), and the expert user who wants to enter the command in abbreviated form can do so without delay. For the novice user, typing escape serves several purposes:
Confirms that the input entered up to that point is legal. Conversely, if the user had made an error, he finds out about it immediately rather than after investing the additional and ultimately wasted effort to type the rest of the command.
Confirms for the user that what the system now understands is (or isn't) what the user means. For example, if the user types DEL, the system completes the word DELETE. If the user had been thinking of a command DELAY, he would know immediately that the system had not understood what he meant.
Typing escape also makes the system respond with any "noise" words that may be part of the command. A noise word is not syntactically or semantically necessary for the command but serves to make it more readable for the user and to suggest what follows. Typing DIR and escape actually causes the display to show:
DIRECTORY (OF FILE)
This is a key underlying reason that the TOPS-20 interface is perceived as friendly: it significantly reduces the number of large negative feedback events which occur to the user, and instead provides many more small but positive (i.e. successful) interactions. This positive reinforcement would be considered quite obvious if viewed in human-to-human interaction terms, but through most of the history of computers, we have ignored the need of the human user to have the computer be a positive and encouraging member of the dialog.
Typing escape is only a request. If your input so far is ambiguous, the system merely signals (with a bell or beep) and waits again for more input. Also, the escape recognition is available for symbolic names (e.g. files) as well as command verbs. This means that a user may use long, descriptive file names in order to help keep track of what the files contain, yet not have to type these long names on every reference. For example, if my directory contains:
BIG_PROGRAM_FILE_SOURCE
VERY_LONG_MANUAL_TEXT
I can excuse shell languages because they're optimizing for being fast to type in a command line, while also being used for writing programs. But other than that, I weep when I think of the time being wasted on languages with arcane syntax. I wish the world would just switch over to s-expressions.
I understand that many people will find well thought out syntax cleaner than writing trees. But what languages like Bash or CMake, and arguably JS too, have in common is that their syntax wasn't thought out, it grew organically into a mess. S-expressions eliminate this entire class of problems up front.
[ $X == "hello" ]
example.sh: line 5: [: ==: unary operator expected
[ x$X == x"hello" ]
[ "$X" == "hello" ]This could be either because the glob ends up unintentionally matching something:
$ var='*'; touch xfoo xbar
$ [ x$var == x"hello" ]
bash: [: too many arguments
$ var="Rocky_[1976].jpg"; touch xRocky_7.jpg
$ [ x$var == x"Rocky_[1976].jpg" ]
(false, should be true)
$ var="Rocky_[1976].jpg"; shopt -s nullglob
$ [ x$var == x"hello" ]
bash: [: ==: unary operator expectedDisclaimer: I am one of the current sponsors.
I've puttered along for years writing little bash scripts for one thing or another and always had the uneasy feeling that I was littering them with errors.
Shellcheck's linting showed me (a) that I was right and (b) how to fix them. It's made me a better developer and I'm so grateful for it!
sh.ls("-l", "/tmp", color="never")Now there will be tones of angry dudes complaining about:
- Optional verbosity
- 1-5s startup time on some machines
- Random MS agenda to conquer and destroy
- Random problem with usage that is solved by RTFM
- Suggesting python, ruby, go, node, perl and whatever non-shell language
Thankfully, ignoring is still a great human feature.
However, you grossly overestimate the “awesome”. To answer each “angry dude” point in turn:
- The verbosity is fine - in scripts, one should fully expand command names. Interactively, `gci` is fine (or the `ls` alias, though some of the default aliases are travesties).
- The startup time is insane. I start a new terminal window hundreds of times per day, and the startup time is what made me abandon POSH.
- Apache 2 licensed, so I don’t care. I do default to assuming that MSFT are doing something shady though, and I’m right [1].
- RTFMing is perfectly fine (and Get-Help has high quality documentation for built-ins), but one must accept that most users will not. ISE (if it existed outside of Windows) was a nice workaround for needing this.
- Python, Ruby, Go, Node and Perl are all more portable (see my last comment about FreeBSD support).
As an interactive shell, nushell [2] is probably the closest thing to the PowerShell experience which is not tied to .NET Core.
[1]: https://docs.microsoft.com/en-us/powershell/module/microsoft... [2]: https://github.com/nushell/nushell
As far as I can see, the only really problematic one is startup time, which is surely something that will be resolved in the future. I witnessed this only rarely while I use PowerShell on Windows and Linux - it was on windows and it was on VM with very slow disk, and some modules that each keep functions separated by files.
I welcome nushell development, but you can't really compare it to pwsh - even if they continue developing it, its decade or so from being usable as main shell.
> you grossly overestimate the “awesome”.
Nah, its awesome. I save bunch of time any time I open the PowerShell. And any time when one tries to be quicker in some other language one is not, eventually.
> [nushell is a] decade or so from being usable as a main shell
News to me. I replaced Powershell with NuShell.
cross platform pwsh is just few years old
> I replaced Powershell with NuShell.
Sure, you can replace it with LOLCAT either, its totally legit.
Regarding startup time, I can't believe there is such drama. If that troubles you so much, you can easily workaround it by always keeping powershell process pool ready in the background (1 or 2 suspended instances until you use them).
It's slow on Windows too.
This is a pretty absurd take when the discussion is about alternative shells for writing scripts. Shell scripts are everywhere. A fairly heavily used UNIX box might easily execute hundreds of shell scripts every minute. A startup time of several seconds makes it completely unusable.
Pattern of usage of bash scripts where hundreds of shell script run every minute is IMO not something you should brag about in systems architecture. And pwsh is different - it has modules which are encapsulated and you don't have to start another instance of shell to prevent mingle. Parallelism ofc still benefits from faster startup.
OK. So then startup time is not slow? Well, then it's of course a non issue - but then I also don't see why a process pool would be required.
> Pattern of usage of bash scripts where hundreds of shell script run every minute is IMO not something you should brag about in systems architecture.
Yet, you just did :) Anyway, it was hardly intended as bragging, rather than just stating facts. Lots of stuff are shell scripts. Lots of user facing commands are wrapped in shell scripts. Of the ~3k commands in my $PATH, around 500 are shell scripts. You may not think this is a good state of affairs, but it's still a fact. (And it's quite unclear to me what's so bad about it - an excellent use for scripts is providing environment for other programs.)
Its not as fast as bash or cmd but I don't start hundreeds of instances of shell
> but then I also don't see why a process pool would be required.
If you have habit of starting shell every ten seconds or so and equally fast close them, I guess powershell will not be your friend. Bad habit to have anyway IMO, because there are few tools that support that workflow.
> Yet, you just did :)
I never said I run bunch of instances of pwsh - I usually run couple only and within them, everything happens. Although pwsh starts in 300ms on my machine.
> And it's quite unclear to me what's so bad about it
The bad about it is jungle of bashizms and next to 90% of statements which have nothing to do with actual business logic but are there for text massage and parsing.
You can't really compare bash and pwsh tho - you have entire dotNet ecosystem there compared to bash where you need jungle of tools. You can do almost ANYTHING in pwsh without relying on external tools. That is consistency - on different flavors of linux, some tools are there, some are not, and some are just different, even the basic ones such as stat. I get nightmares from the fact that grep and sed and awk and perl and rg and whatever each use their own flavor of reg ex... Its just insane :)
pwsh is more comparable to mainstream languages like go or python. What separates it from them is that it is designed for shell. With that power, I can understand a little bit of slower startup time, but I am sure it will be solved eventually.
This is absolutely false. Powershell and the Windows console prompt do not support this workflow. Every other shell/terminal emulator combination I can think of supports it fine.
Perhaps you can elaborate on why you consider it to be a "bad" habit?
Others I've not used are oil shell, which aims to be a better Bash. Elvish, which has a heck of a lot of similarities with my own project (and is arguably more mature too). There are REPL shells in LISP, Python and all sorts too.
So there are options out there if you want a $SHELL and happy to break from POSIX and/or bash compatibility.
Extra-long command names and deeply namespaced .Net types can be a chore both to read and write, but I find the *nix tools way too cryptic. Never could get used to them, but then again, I've been on DOS/Windows all my life since the 90s.
To be honest, I'd occasionally get frustrated and think about switching to Linux. However, Python and PowerShell are the two things that have kept me on Windows. The fact that they are both open-source and cross-platform is just the cherry on top.
For some reason it is commonly believed that writing shell so easy that one don't need anything of this. Don't even need to learn it.
However, good systems impose elegance without users having to create complex standards around it.
And C is still omnipresent. So there must be some other characteristic of Bash that makes it not fit for this age (or some characteristic of C that makes it irreplaceable, that Bash does not have)
And actively being replaced by rust each day because those "standards" are obviously not enough.
And there is some other characteristic of C - its portable.
May be availability of the shell is an unfortunate historical artifact, yet it still the reason why one may want to use it.
2. Idea to impose elegance upon users is attractive, but even if we ignore that it makes system/language design harder, different people can have different option on what is elegant and what is not. Having said that I like Go (which tries to enforce what can be enforced) because options of its author are not too far from mine. But not everybody is happy with Go and it is not hard to find its criticism.
Thera are also measurable effects of elegant system like number of bugs, questions, libraries, standards, etc.
... would fail to run test commands whose left-hand side matched a unary operator.
This was fixed [...] in 1979. However, test and [ were also available as separate executables, and appear to have retained a variant of the buggy behavior:
Often they're things you will get away with most or even all of the time in a specific scenario, but there they lurk, waiting to blow your leg off when you start including spaces in a filename or somesuch!
What resources would you suggest for decent training material?
What I've done is, every time I came across something that didn't behave the way I expected, I tried to Google it and tinker around with it until I could understand exactly what is going on, so that I "got" the underlying feel for the language and didn't have to look up every consequence every time. e.g., the difference between [ ] and [[ ]], which is confusing at first, becomes more intuitive once you realize [ is a portable POSIX command whereas [[ is special Bash syntax, and once you grasp this, you can understand what its implications are. e.g., that Bash has no way to detect '=' as a special token inside [ ], and consequently you should worry about ambiguities. (This does assume a little background knowledge of parsing ambiguities; if you haven't taken a course related to programming languages, going through one might also be a worthwhile investment.)
Also, you have to be on the lookout for things that you would want to do in a different language, and learn how to do it in Bash, like creating functions & local variables. It's mostly a matter of being proactive about learning how to structure your code like you would in other languages. Often I see people try to keep things "simple" by just copy/pasting code so that they don't have to organize things at a higher level. You wouldn't do that in other languages; don't do that in Bash.
Some pitfalls you really just do have to learn the hard way; there's no other way I can think of. The key here is to spend the extra minutes necessary to actually learn what broke when you come across it, instead of just tinkering with your code until it works. Like when you realize that (set -e; false && false; echo hi) prints 'hi' despite the fact that (set -e; false; echo hi) does not (this is absolutely bonkers), dig in and figure out what's going on so you don't shoot yourself in the foot the next time. You can't avoid it the first time, but you can at least make sure to avoid it in the future.
Also, always try to think of unintended consequences when you learn how to do something in Bash. Like when you learn how `trap 'rm -f file.bak' EXIT` lets you clean up a file, ask yourself what happens if someone has already set a trap? Ask yourself how to dynamically generate and pass a file name to that command safely? Ask yourself in what scope/context the command is evaluated? Spend some time figuring these out either online or by tinkering manually, then remember your solution for later (maybe write it down). Possibly the most costly mistake you can make is to assume something will work in exactly the same way you expect in every other language, because chances are pretty darn good it won't. Even pipelines (which you probably already use) are tricky if you start thinking about them... e.g., what happens if the first command terminates prematurely? What happens if you assign to a variable? etc.
Some sites I can recommend for some things:
- https://pubs.opengroup.org/onlinepubs/9699919799/utilities/s...
- https://tldp.org/LDP/abs/html/index.html
- https://mywiki.wooledge.org/BashPitfalls
The POSIX pages (OpenGroup, first link) are useful for understanding what you can rely on; I'd use that as a command reference whenever it's possible. In some cases they're overly pessimistic (e.g., you can probably rely on 'local' despite it not being POSIX), but generally they're good starting points.
tldp is where I still look up some things, like the heredoc syntax. I've probably looked that up 20 times and I still don't remember it well enough for some reason.
And it might be fun to just browse through the BashPitfalls page once you have some comfort with shell scripting just to get an idea of what can go wrong, but I wouldn't really recommend it as a learning guide, and you definitely shouldn't expect to memorize all of it.
(Usually available locally at 'info bash', I know you hate info don't @ me)
The one resource that I have bookmarked decades ago and would highly recommend to get the ball rolling are the TLDP pages on bash:
https://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
https://tldp.org/LDP/Bash-Beginners-Guide/html/index.html
https://tldp.org/LDP/abs/html/
Not certain if they’re up to date or error free but I really enjoy the style and tutorial/show by example nature of the TLDP
1. The REPL is top notch.
2. Good documentation. Most commands come with a complete manual and examples are easy to find.
3. It's portable and has lots of available APIs, probably even more than C.
Right when this happens, that very second, remember there are hundreds of other languages that can do anything bash can in a readable fashion. EG. os.system() in python. If you are writing more than 3 lines of bash you are doing it all wrong.
(PHP has its oddities, but it has many benefits over Bash, including an optionally strictly-enforced type system, binary-safe strings, clear C-like syntax… and like Bash it has backticks!)
(ba)sh scripting has so many footguns it astounds me that so much complicated code has been written (more-or-less) successfully in it. Once I reach more than a few dozen lines I usually hit the escape hatch and reach for a more powerful scripting language (you can always still shell out to utilize handy CLI tools if you need them).
[1] https://www.newline.co/courses/newline-guide-to-bash-scripti...
Maybe it is time to consider using rust instead: https://github.com/rust-shell-script/rust_cmd_lib ?