Bash functions are better than I thought
cuddly-octo-palm-tree.com
cuddly-octo-palm-tree.com
If you use Bash for programming, you have to stop thinking in terms of the holier-than-thou software engineer, whose ego believes that a superior, "clean" design makes a superior program. You should embrace globals. You should switch between using or not using the enforcement of set variables or program exit status. You should stop using Bashisms and subtle, obscure language features unless you absolutely have to.
Bash is not a "real" programming language, so do not treat it as one. Do not look for hidden features, or try to do things in cute ways that nobody else uses. There is no superior method or hidden knowledge. Just write extremely simple code and understand the quirks of the shell.
On the other hand, I wonder if the web page keeps track and could offer "best of bash mistakes".
But if you're still using Windows, then yes, I agree.
"ShellCheck is...
* GPLv3: free as in freedom
* available on GitHub (as is this website)
* already packaged for your distro or package manager
* supported as an integrated linter in major editors
* available in CodeClimate, Codacy and CodeFactor to auto-check your GitHub repo
* written in Haskell, if you're into that sort of thing. "
Sometimes you are not in your computer, the script does not have private information (i.e. open source or something you don't care to be public). Sometimes the website is simply more convenient.
*edit: formatting
https://github.com/GavinRay97/hasura-ci-cd-action/blob/a9731...
This is out of necessity. I'm not the sharpest tool in the shed, so I have to go out of my way to write things such that when I come back to them in months or years, I still understand what they do.
For this same reason, is also why I never use shorthand flags for scripts.
I have no clue what IE: "-s -y -o" might do (or even worse, "-syo", dear god my eyes! Also -- is that one special command, or a series of individual commands?!).
But "--silent --assume-yes --output-file" is pretty easy to grok immediately.
Oh man this is so ugly
Not something I would expect in an "elegant" script. Don't think that it is your fault, just it is very hard to write anything elegant in bash.
Hahaha -- fair point. Readable: maybe. Elegant? No. Bash is far from elegant =P
Sorry if I missed something in the logic, reading on my phone, but from the comment, this feels like something I do frequently...
The thing with bash is, it's great for tying things together and quick bits and pieces, but it's not set up for writing maintainable code. Arrays, functions, even if statements comparisons can all be done in bash (as first-class features), but are just... easier in other languages. And then think about the refactoring, linting, testing tools available in bash vs other languages. And then on top of that, there's the issue of handling non-zero return codes from programs you call; do you `set -e`, and exit on any non-zero return code even if you wanted to continue, or not `set -e`, ignoring any errors as your script just continues.
Personally, when I feel I want to use a function (or array, or other similar, non-trivial thing), in bash, it's time to reach for another language.
Having said that, there are some nice programs written in bash. https://www.passwordstore.org/ being one that comes to mind.
Then you must not be writing any bash at all. Functions are useful for almost anything beyond a one-liner script.
Typical functions used in innumerable non-trivial scripts:
* print_usage()
* die() (see: https://stackoverflow.com/q/7868818/1593077)
:-)
With regards to `print_usage()` and `die()`, yes, I would reach for Python 3 then. The `argparse` module and `throw` are first-class members of the stdlib/language and are better and more standard between programs than if I threw together these myself (and with `throw` you get a stack trace, which is nice).
Python is strongly typed. Maybe you meant "statically"? (As opposed to dynamically.)
https://ilya-sher.org/2021/03/19/running-elegant-bash-on-sim...
Parsing command-line arguments is easy, but parsing them correctly is ridiculously hard. `getopts` doesn't handle long flags, so it's out. `getopt` doesn't handle arguments with spaces in them, unless you have a version with non-standard extensions, so it's out. You're left with manual parsing, and it's a royal pain to make sure you handle every expected case (short/long flags, clusters of short flags, trailing arguments to be passed through to a subprocess, short/long options whose value is in the next argument, long options whose value is after an equals sign, and several others I'm probably forgetting about). And this is just to get the arguments into your script, before you actually do anything with it.
I agree that bash has effective idioms, and learning those idioms can make scripts easier to write. I strongly disagree that bash is "as maintainable as anything else", and scripts beyond a few hundred lines should be rewritten before they can continue growing.
Is there a feature I'm not seeing that would generate an argument parser without external dependencies?
...
--foo)
FOO="${2:?ERROR: String value is required for \"-f|--foo\" option. See --help for details.}"
shift 2
;;
--foo=*)
FOO="${1#*=}"
shift 1
;;
... while getopts :i:h-: option
do case $option in
# accept -i $input-file, -i$input-file
i ) input_file=$OPTARG;;
h ) print_help;;
- ) case $OPTARG in
help ) print_help;;
help=* ) echo "Option has unexpected argument: ${OPTARG%%=*}" >&2; exit 1;;
in=* ) input_file=${OPTARG##*=};;
in ) echo "Option missing argument: $OPTARG" >&2; exit 1;;
* ) echo "Bad option $OPTARG" >&2; exit 1;;
esac;;
'?' ) echo "Unknown option $OPTARG" >&2; exit 1;;
: ) echo "Option missing argument: $OPTARG" >&2; exit 1;;
* ) echo "Bad state in getopts" >&2; exit 1;;
esac
done
shift $((OPTIND-1))Since all valid "sh" syntax is valid "bash", you've got to restrict yourself to only sh-valid code in the event the host doesn't have bash.
I have gotten used to doing this -- these scripts usually wind up being run in Docker containers that don't have "bash" installed for the sake of minimalism.
IE, you can't use [[ $predicate ]], but instead [ $predicate ], etc. Lot of subtle differences.
The supersets of POSIX are often even more effective.
Ugly as the tool may be, in the box it stays.
Bash is definitely a different paradigm to your average imperative or functional language and thus requires a different approach but I wouldn’t go so far as to say it’s not “real”.
It certainly fits the criteria of a programming language even if it does have more warts than a pantomime witch.
Compared to other languages, Bash can be incredibly worse for some things, but much better for others.
People who don't realise that won't be able to get the best out of their toolset.
No. It’s still an interesting yard stick. It just doesn’t describe programming languages in full. For example, and someone counterintuitively, it is actually possible to design a programming language that isn’t Turing complete. Some esoteric functional languages do actually fall into that category. But generally programming languages would be a subset of Turing complete software.
> And on that same note, have you seen the stuff being done in Minecraft?
I have, hence why I cited it as an example.
> It is quite up for debate whether it is a programming language.
One could debate anything but it doesn’t mean the argument is made in good faith.
Minecraft is a game. I’m happy to even extend the definition and say it’s supports some visual programming mechanics. But that doesn’t mean it is a programming language.
Just because something has 4 legs and a tail, it doesn’t automatically mean it is a dog.
Is there a reason we aren’t using a shell with a proper programming language for scripting?
I can’t say I’m a fan of bashscript. I wonder why we are using this weird language.
Maybe it'll be like the C++ vs Rust situation?
After all, HTML got much better when WHATWG took W3C's toys away and actually started innovating HTML again. Maybe it's time to do something similar with Unix: I'm sick and tired of people telling everyone that we have to stay in some 1995 time capsule because we need to follow The Standard and if The Standard hasn't changed, too bad.
De-facto standardization has at least brought us free officially-supported options for running Bash or other *nix shells on both of those platforms(if terribly outdated on Mac).
$ cat "file.txt"
$ ls "*.txt", ["l", "a"]
$ ls("*.txt", ["l", "a"]) > "file.txt"
Also powershell has no support for automatically handling external shell commands that error, you must manually check $LASTEXITCODE (easily done with a custom invoke api that you use for everything).
Also well, some of the escaping in strings is really odd...
With bash (and I suspect with every other popular shells out there), it actually means something different. In your second example, if you don't use quotes, the shell will do the extensions, but if you do, the process will have to do it (and in the case of ls, it cannot).
In some case it's good to have the shell taking care of it, but sometimes (eg, when using 'find') it's not.
I was trying to point out that quoting parameters or not could bring a different meaning for a shell (as one could see it with Bash), with the implied consequence that one would have to carry over that nuance somehow to a shell where all parameters would have to be quoted.
However if I’m not careful they get painfully complex and I tend to regret not writing it in perk. As such I now switch to using Perl when I get to the function stage, or anything other than the most simple bash arithmetic.
You have to step back and remember the point of all this. Why are you programming? To solve a problem, and make a human's life easier. What is the problem you are trying to solve? How can we make the human's life easier? In this case, it's "I want to combine bunch of programs together in a command-line interface, to make it easier to use these programs to get my work done on the command-line." So, what solution should we choose for this scenario?
Lots of different "scripting" languages exist (we used to call them "glue languages"). Today Python is the most popular, and before it was PHP and Perl. But who cares if they're popular? What are they good for? Why would you choose one over the other?
Python is a general purpose language which is easy to learn, easy to write, and easy to read. Well that sounds nice, but it's very "general" and not specific to our use case. PHP is a language designed for use in web development. Perl is a language designed for system administration-type tasks.
Bash scripting is designed for making it easy to combine already-existing programs in a command-line shell, with no modification or creation of programs required. The only "dependency" is the Bash interpreter (which happens to also be the command-line interface we started our problem with! how convenient!) and an existing collection of software tools that work with each other through a command-line interface. Every part of it is explicitly not designed to "make programming easier". Instead, it is designed to make "combining programs in the command-line" easier. The result is things like every single word you type is potentially just an outside program being executed, or arguments to that program. No other language has this quirky behavior, because they're not designed solely to make command-lines easier to use.
I know Perl well, and the language is fantastically useful as a system scripting language. It combines some of the best features of common shell scripting programs with more programming language-specific features. It is designed with shortcuts and "magic" to make quick work of command-line scripting tasks. But ultimately Perl was not built into the command-line, so its utility is always a bit stilted. Going between the shell and the Perl code and back to the shell isn't as flexible as if the Perl code was embedded in the command-line. Perl also doesn't have as simple a facility for pipes as a command-line shell does. Oh, and a Perl install is rather large, isn't available everywhere by default (anymore), and has all the usual dependency problems. So even though I know Perl very well, if I need to just combine existing programs in the command-line, I always use Shell scripting instead of Perl. (There are Perl shells which solve this problem, but then everyone needs to learn Perl, and Perl can be.... idiosyncratic)
You could also use Awk for a large number of general programming tasks, but again it's designed for a different specific problem: pattern-scanning and processing, not generally combining programs together.
So we use shell scripting to solve the problem because it was designed specifically for our problem, it's ubiquitous, and simple to use. You could use any other programming language, but it wouldn't fit our scenario as well. Once your use case changes, and you are no longer only trying to combine existing programs together, or the existing programs don't do what you want them to do, then you need to use a different language that better fits that new use case.
Here are the ways to run a program in ruby https://stackoverflow.com/questions/7212573/when-to-use-each...
There are two very distinct places where Bash is not, that I have found.
Bash is not in busybox. It pretends to be, but what is really there is the Almquist shell, with a bit of syntactic sugar to silence the common complaints.
Bash is also not in Debian's shell (/bin/sh). In that place, there is no tolerance of bashisms.
It is important to know the POSIX shell for these reasons.
The Android userland explicitly avoids any GPL software.
The Android system shell is the MirBSD Kornshell (mksh).
This is (obviously) a massive platform.
When writing a bash script, shouldn't you be setting the shebang to /bin/bash which is available on Debian?
It is otherwise advisable when speed or portability are required.
I've still not found a suitable answer.
https://research.swtch.com/shmacro
The Bourne shell thus adopted ALGOL syntax, distinct from anything else in UNIX.
David Korn brought a pile of C, and wedged it into a max 64k program space (for Xenix), and his upward-compatible Korn shell (ksh88) had many, many features.
Then there were the "UNIX Wars"...
https://en.wikipedia.org/wiki/Unix_wars
The fallout of this was the choice of a few Korn features, but not all (no arrays, coprocesses, [some] eval, regex, et al.)
The wars defined the POSIX shell. Yes, it can be infuriating.
Mostly because the people who want to introduce a "programming language" into the shell don't prioritize being a shell.
Check out the "Oh" shell for contrast. This is what a programming language looks like when you force it to conform to being a shell first priority.
https://github.com/michaelmacinnis/oh
https://www.youtube.com/watch?v=v1m-WEZz46U
This is "Scheme-like" (actually the late John Shutt's Kernel-like) but has FEXPRs so things can be redefined and evaluation can be controlled.
djb has always used them, even in the shortest of scripts.
Otherwise I agree with everything in this comment.
There is a reasonable argument to avoid using bash for non-interactive scripts. The benefits of any additional bash features, so-called "bashisms", arguably do not outweigh the costs of making these scripts non-portable. One of the many advantages of shell scripts is that they are portable and have tremendous longevity; shell scripts can last a long, long time. There are no version changes and aggressive feature creep to worry about as is routinely the case with programming languages. The scripts just keep working, every day, and we can forget we are even using them, e.g., they are being used in the various ways by UNIX-like OS distributions.
One of the today's programmer memes was "Get Stuff Done". Maybe the shell was not meant for today's programmers. But for "sysadmins", or "DevOps", or whatever term anyone comes up with in the future, people who can "administer/operate" computers running a UNIX-like OS for themselves or for someone else like a client or an employer, the Bourne shell works for its intended purpose, better than anything anyone has come up with in the last 50+ years. There's a lot of stuff that "gets done" with the shell, whether it is on someone's own computer, their client's or their employer's.
Attempts at "shell replacements", cf. alternative shells, usually look like interpreters for programming languages, not shells. Perhaps this is not a coincidence.
Bourne shell is relatively small and fast. Computers with small form factors often use UNIX-like OS and when they do they usually include shells. Many programmers seem to dislike the notion that such a layer exists. They often try to blame the shell instead of their own lack of interest in learning to use it.
The shell is boring, and "boring" is sometimes the wisest choice. Most software has expanded to consume available resources (often the developer's choice not the user's). This makes computer performance gains difficult for the end user to discern, e.g., decade after decade, routine tasks seem to take the same amount of time. However the shell and many "standard" UNIX utilities have not changed as much in the same period. Subshells may have been "slow" many years ago. IMHO, they do not feel slow today. Routine tasks performed with the shell seem to run faster, as one would expect after hardware upgrade.
I posit: Life is too short to learn every programming language du jour but it is long enough to learn the Bourne shell, reasonably well.
They have to shell out to other commands in order to do something meaningful.
Depending on your OS, these commands may not exist, or have a different set of commandline flags.
If scripts written for a POSIX-like "lowest common denominator" shell were not portable, methinks projects written for such a shell, such as autoconf, would not work on so many diffferent systems. The amount of free software found in open source OS repositories that is built using "configure" shell scripts is not small.
Debian and other Linux distributions use a shell derived from NetBSD's Almquist shell. People who design and maintain these operating systems suggest that, for non-interactive use, this Bourne shell is faster than Bash.
I'll count myself among nobodies, I guess. I use them a fair bit when part of said subshell cd's somewhere, and I don't feel like using pushd/popd, either of which might fail to operate based on my mistakes, but subshells seem to never fail to exit, eventually.
Mind, if possible, it is even uglier than using stdout.
err_f="$(mktemp)"
third_f="$(mktemp)"
out="$(cmd 2>"$err_f" 3>"$third_f")"
err="$(cat "$err_f")"
third="$(cat "$third_f")"
The fact that there isn't a good way to open an anonymous pipe and get its two file descriptors in bash is very annoying.
Edit: It seems I should elaborate.
> Getting anything out of a subshell that isn't from STDOUT is impossible. So you can't define an array in a subshell and then use it outside the subshell, and you can't return an array (or anything that isn't a string) from a subshell. If you only use subshells and want to use any kind of data structure that isn't a string passed from STDOUT, you have to do it globally.
Yes, but that is no problem at all, because that is the way shell scripts work and if you do it in a functional way, where the function has no state, it is great (we a few exceptions).
> And subshells are slow. So nobody uses subshells.
Shell scripts are slow in general, subshells don't make an exception but also don't have a large impact. And people do use subshells.
> If you use Bash for programming, you have to stop thinking in terms of the holier-than-thou software engineer, whose ego believes that a superior, "clean" design makes a superior program. You should embrace globals. You should switch between using or not using the enforcement of set variables or program exit status. You should stop using Bashisms and subtle, obscure language features unless you absolutely have to.
If you use Shell scripts, you should understand that this language has been designed decades ago and that professionals advise to use it just in short scripts to connect binaries.
> Bash is not a "real" programming language, so do not treat it as one. Do not look for hidden features, or try to do things in cute ways that nobody else uses. There is no superior method or hidden knowledge. Just write extremely simple code and understand the quirks of the shell.
That part is mostly okay, but "real" doesn't get the point, as it is a real language, just one with many problems. However, given its superior strength we haven't overcome it yet...
"Superior strength" is probably just a matter taste, however I would really like to read your explanation on why you think so.
Some reasons why POSIX shells are great:
- they are part of POSIX (a Standard which is implemented by multiple OS) and therefore often available
- you can use them to compose programs written in different languages
- you can easily connect the output of one program with the input of another (pipes)
> they are part of POSIX (a Standard which is implemented by multiple OS) and therefore often available
That's not an intrinsic property of Bash.
> - you can use them to compose programs written in different languages
> - you can easily connect the output of one program with the input of another (pipes)
This last point subsumes the previous one. So basically, you're saying Bash is great because it's a shell?
myFunc() { (
#...
) }
1. Return values of subshells 2. Clean design 3. being a "real" programming language
The only Bash specific part was
> You should stop using Bashisms and subtle, obscure language features unless you absolutely have to.
But in fact, the usage of subshells to limit the scope and subshell return values have nothing to do with Bashisms.
Also, JavaScript really is not a very good programming language. We are just stuck with it because it's the only language the browser understands. (Well, until recently: things are going to change with the introduction of wasm).
But for the shell, we're not stuck with any one language. Whatever you want to do can be programmed in your favorite language. You can easily write a python script instead of a bash function.
people have been saying this for years already and nothing seems to have come of it :(
We've been forgetting and relearning that for the past two decades.
Actually it turns out that having an interpreted runtime causes a stream of endless problem. We keep try to patch them over but new problems keep coming up.
It turns out that a standarized low level byte code is the right way to ship cross platform applications. We already knew that. It just took a long time to go through the standarization process and be implemented by browsers.
If it seems like it's not in use today, it's largely because JavaScript has momentum and the majority of web programmers don't yet know how to take advantage of wasm (or maybe don't think they need it).
So why hasn’t it taken over the world with Java and now with wasm?
> If it seems like it's not in use today, it's largely because JavaScript has momentum and the majority of web programmers don't yet know how to take advantage of wasm (or maybe don't think they need it).
There’s another possible reason. I’m sure you can come up with the answer yourself.
Java applets really sucked. They took a long time to load and initialize. They all looked ugly (probably because of the default libraries?).
Also Java itself as a language was really bad and the development experience was awful. I want to say that no one wants to program in Java but in reality many people do (I don't understand those people).
The important lesson here is the implementation is more important than the idea.
Good idea with bad implementation -> goes no where.
wasm is not java.
The important quality about wasm is that it's not garbage collected. It's pretty close to just good old assembly.
Chrome was released at the end of 2008, JavaScript was thriving way before that. We had Gmail since 2004, jQuery since 2006. WebApps were the “sweet solution” considered for iPhone apps at first in 2007. Chrome exists because of the healthy ecosystem that Firefox and Safari provided, not the other way around.
>It's pretty close to just good old assembly.
Precisely, and it's yet to be proven that's the best solution for the Web. I love it from the computer science perpective, but historically that idea hasn't struck a chord.
I thought that was the case but I can’t recall where I read it. If that’s not true then I should probably reconsider my thinking about wasm
There aren't many languages that are practical for WASM output. Scripting and managed languages need to ship their interpreters and runtimes with their WASM blobs, and can end up relatively large. JavaScript's interpreter and runtime are baked into every browser already.
That leaves only compiled and unmanaged languages for potentially good WASM targets. As mentioned before, Rust is seeing a lot development in that space. If LLVM can compile it, then Emscripten can output it to WASM.
Figma for example seems to be largely powered by WASM and the core of the product seems to be written in C++
https://www.figma.com/blog/webassembly-cut-figmas-load-time-...
https://www.figma.com/blog/building-a-professional-design-to...
I think the majority of web developers are just slow at catching up with this new things.
Its tooling got significantly better in the last decade or so, but you still need to intentionally rely on it in order to work with the language in any sensible way. You can't really hope for any other outcome when maintaining backwards compatibility.
Everything else is either unmanageably complicated (webpack) or slow (rollup, parcel, babel).
React mainstreamed the idea of using a virtualdom to accelerate UI building (accelerate in the computational sense: by not dealing directly with the slow DOM).
> JavaScript really is not a very good programming language.
Go look at ES6 (the javascript version that came out in 2015) and even TypeScript. You'll be pleasantly surprised, no matter if you're coming from C++, Java, C# or even other "scripting" languages such as PHP and Python.
Ultimately I want a langauge with value types like structs and arrays that you can use to do computations without allocating things on the heap.
In Go I can write a function that returns two numbers.
In Javascript I can only return two numbers by allocating an array on the heap with two items. These two items themselves are probably pointers to two number objects allocated separately. So that's 5 allocations to return two numbers. It's insane.
This means fundamentally it's impossible to create applications in javascript that are both sophisticated and high performance.
A great example to illustrate this is the tooling around Javascript itself.
There were all kinds of bundlers: webpack, rollup, parcel, etc. Rollup was considered the fastest and lightest. They were all programmed in Javascript itself. They were all slow. But no one really knew how bad it was because there was nothing really better.
Then esbuild comes along, and blows all of them completely out of the water, out performing them by 100 times. And it's written in Go. A language that supports structs and arrays as value types.
For example, you are worried about value types but under the hood the JIT compiler will actually generate efficient representations that are passed by value if you do not mess things up for it. Modern JS compilers are extremely sophisticated.
For returning values without an allocation, you’d create such a “struct” and then use a memory pool. It would still be heap allocated I believe but not totally sure. People who know more than I do could probably tell you a better method to return multiple values from a function efficiently.
Yes, and in my experience this kind of applications (and the browser in general) are one of the only things these days that still push me to upgrade my hardware to have better CPUs and more RAM, as it's perfectly sufficient as is for pretty much everything else I do.
A couple of years ago, I challenged myself to write some complex scripts and apps using only Bash, and came to this conclusion myself.
You can get pretty far with just Bash alone, especially if you strive to write readable and maintainable code, and not just one off scripts. If you need access to data structures, especially nested ones, you can shell out to another language and then print the results back to stdout so Bash can use them:
$ get_data | transform_data | python3 ./complex_script.py | consume_resultsPackaging in Python is a bit more of a pain.
> Windows doesn't have bash preinstalled
> On all OSes I've used in the past 3-4 years all of them have been Python3 by default.
I just hate doing things with files in the shell, all those stupid escaping rules, and god forbid you have files with spaces or quotes in the names, or leading or trailing spaces.
It's just spending time learning hacks to get around shell limitations instead of actually getting things done, and at the end it's an ugly mess.
Or maybe I'm just bad at it, but I don't think it's only that.
should read
"no reason to think ... couldn't"
> avoid bashisms
What's holier-than-thou is insisting that people stick to a dialect of shell scripting that hasn't changed in decades because some people think that conforming to POSIX is its own reward even if it makes life harder and programs worse.
No, thanks. I'll stick to bashisms. Things like coprocesses and filemap make certain classes of program much easier to write. And yes, shell scripts are programs, and the quickest way to turn these programs into unmaintainable balls of mud is to follow your advice to avoid normal programming best practices simply because the program you're writing happens to be a shell script.
#!/usr/bin/env bash
process_line() {
if [[ $* == *foo* ]]; then
echo "found_foo=1" >>$IPC
fi
}
IPC=$(mktemp)
find . -maxdepth 1 -type f | while read line; do
# note: this is run a subprocess
process_line "$line"
done
source "$IPC"
echo "found_foo=$found_foo"Huh. How does one return a value from a bash function? I've always only ever seen the "echo calling convention" used in scripts, or use of global variables.
You can use I/O redirection with arbitrary file descriptor numbers, and you can let the shell pick them. So you can `mknod` a pipe, `exec {pipefd_rw}<>"$pipe_name" {pipefd_ro}<"$pipe_name" {pipefd_rw}>"$pipe_name";` and now you have three open file descriptors that you can let the sub-shell use to communicate with the parent. And so you're not limited to stdin/stdout. I do wish there was a built-in for making an anonymous pipe instead of having to make one on filesystem.
(You have to open the named pipe for read-write first because otherwise you'll block, but if you don't want a read-write FD for it you can close it after opening the read and write FDs.)
If anything you are saying it should be deprecated in place for a real language. If people want to do more advanced things why does the basic language of their shell not support it? Why should you not care about the correctness of your shell scripts which can run important things? Seems a perfect place for a functional language. It’s high level, you can just declare what you want. I imagine your interpreter can add in all sorts of checks to help make your scripting more correct.
And it’s a horrible shell experience.
It’s great for scripting (but not as good as other scripting languages) but it’s a terrible experience on a day to day basis, with the only salvation being the existence of 2-3 letter aliases which make things slightly more manageable.
Bash is a great shell language that whose scripts are useful because it allows you to trivially convert your manual shell actions into an executable script.
If you need to do anything complex (or rather, something you probably wouldn’t type into the shell manually if you had to do the thing on a one off basis), then you’re probably better off using a different scripting language.
But changing BASH with scripting as a priority would probably make it a worse shell language (which is not to say there are no improvements to be made…there are).
> Getting anything out of a subshell that isn't from STDOUT is impossible.
That said, for scripts whose main input _is_ from a file or STDIN, and also whose main output _is_ to a file or STDOUT, then bash is far more often the right tool for the job than it is given credit for. Of course, I'm talking about bash and a host of other utilities that are often packaged with it such as awk, grep, sed, cut, etc.
For processing text I find myself often choosing between bash and Python, and very often if I choose bash I'll feel that I've made the right choice.
It doesn't trigger EXIT, but it does trigger RETURN. Just trap both:
#!/bin/bash
foo() {
trap "echo 'Cleanup!'" RETURN EXIT
#return
#exit
echo "Kill me with ^C or \"kill $$\""
while true ; do : ; done
}
foo # should print 'Cleanup!' on SIGTERM,
# returning, or calling exite.g.: wtfp.sh
#!/usr/bin/env bash
map() {
local fn="$1"
local input
while read -r input; do
"${fn}" "${input}"
done
}
reduce() {
local fn="$1"
local init="$2"
local input
local output="${init}"
while read -r input; do
output="$("${fn}" "${output}" "${input}")"
done
echo "${output}"
}
filter() {
local fn="$1"
local input
while read -r input; do
"${fn}" "${input}" && echo "${input}"
done
}
add() {
echo $(( $1 + $2 ))
}
increment() {
echo $(( $1 + 1 ))
}
square() {
echo $(( $1 * $1 ))
}
even() {
return $(( $1 % 2 ))
}
sum() {
reduce add 0
}
map increment | map square | filter even | sum
$ printf "%s\n" 1 2 3 4 5 | ./wtfp.sh
56The issue is that pipe mills are very slow.
$ mill() { while read -r line; do echo $line; done }
$ export FIVE_MEGS=$(( 5 * 1024 ** 2 ))
$ time yes | mill | pv -S -s "$FIVE_MEGS" > /dev/null
5.00MiB 0:00:23 [ 221KiB/s] [============>] 100%
real 0m23.084s
user 0m14.121s
sys 0m26.780s
$ time yes | pv -S -s "$FIVE_MEGS" > /dev/null
5.00MiB 0:00:00 [6.55GiB/s] [============>] 100%
real 0m0.005s
user 0m0.000s
sys 0m0.006s
$ export PYLOOP="from sys import stdin
for line in stdin:
print(line)"
$ time yes | python3 -c "$PYLOOP" | pv -S -s "$FIVE_MEGS" > /dev/null
5.00MiB 0:00:00 [67.1MiB/s] [============>] 100%
real 0m0.082s
user 0m0.071s
sys 0m0.019s
dash is faster but not as much as I'd expected:
$ time yes | bash mill | pv -S -s "$FIVE_MEGS" > /dev/null
5MiB 0:00:24 [ 208KiB/s] [====================================================>] 100%
real 0m24.614s
user 0m21.338s
sys 0m11.749s
$ time yes | ksh93 mill | pv -S -s "$FIVE_MEGS" > /dev/null
5MiB 0:00:11 [ 447KiB/s] [====================================================>] 100%
real 0m11.451s
user 0m6.480s
sys 0m12.441s
$ time yes | dash mill | pv -S -s "$FIVE_MEGS" > /dev/null
5MiB 0:00:07 [ 663KiB/s] [====================================================>] 100%
real 0m7.720s
user 0m4.687s
sys 0m8.934s
$ time yes | awk '{ print $0 }' | pv -S -s "$FIVE_MEGS" > /dev/null
5MiB 0:00:00 [11.5MiB/s] [====================================================>] 100%
real 0m0.439s
user 0m0.444s
sys 0m0.011s breaker_breaker() { break; }
foo() { breaker_breaker; }
while true; do
echo Loop
foo
done
Dynamic stack-crawling is more or less the most popular historical behaviour (though afaict the Bourne/Korn lineage just silently fails breaks outside the enclosing function); it's just generally consistent with everything else in the shell that's globally/dynamically scoped. Even in the shells where you can't break out of your enclosing function, `eval break` still breaks through loops in your current context, and that kind of looks like a function you called that called break if you squint.
(POSIX expressly leaves this case undefined with a carveout for break/continue inside the body of a function lexically contained within a loop.)
It also reminds me of TCL's "uplevel": https://www.tcl.tk/man/tcl8.4/TclCmd/uplevel.html
(... unless I miss something.)
subshells are ridiculously slow because it's a forked child process. imagine forking for every function call in your program....
> I simply do not understand why people keep recommending the {} syntax at all.
because it's almost always what you actually want.
Not for built-in commands.
> Yeah “fork is slow” is the sort of microbenchmark that is mostly irrelevant for shell scripts
Maybe if you're on a Linux kernel, but not everywhere else.
For mass processing, I will use xargs to minimize the number of processes created.
#!/usr/bin/env bash
set -eo pipefail
function hey {
echo "Hey!"
}
TIMEFORMAT=$'\nTask completed in %3lR'
time "${@}"
Feel free to replace "time" with "eval" too if you don't want your command timed.
This is a really useful pattern because it means you can create a "run" script with a bunch of sub-commands (private or public), auto-render help menus and create project specific scripts without any boilerplate. Bash also supports having function names with ":" in the name so you can namespace your commands like "./run lint:docker" or "./run lint:frontend".
I have a practical example of this sort of thing here: https://github.com/nickjj/docker-flask-example/blob/main/run
I've written about this pattern in more detail here https://nickjanetakis.com/blog/replacing-make-with-a-shell-s.... It's basically a less limited Makefile for when you want to make project specific aliases. This is something I use all the time now.
My shell is my computing environment. It's rude for a script author to make me change my computing environment when he could just as easily have made a script and created his own sandboxed environment that wouldn't interfere with mine.
People should never ship end user interfaces as bundles of shell functions that have to be sourced.
(I'm looking at you, Android build system.)
If you check the first link in my previous reply it does include an auto-generated help menu that uses compgen to print a list of functions near the bottom. This way you can run `./run` or `./run help` to get a list of commands. I find this is helpful enough without needing completion, especially since every function ends up being a top level command.
Did you mean chmod +x run?
For example, take the command `./run echo "Don't"`. When using `time`, this will print `Don't`. If you use just `"${@}"`, it will also print `Don't`.
But if you use `eval "$@"`, this will crash with an error like
eval: line 10: unexpected EOF while looking for matching `''On the other hand, with a lot of glue work I do, I eventually want to something more complex (use lists and maps, complex string building and regexes, date handling) and while you /can/ do that in bash, I might as well start in Python and have everything in one language and take advantage of things like code sharing via modules. (And yes you can share code in shell, but again it’s not as nice.)
If there's a modern library/workflow that makes this not the case, I'm all ears!
This is exactly how I felt. Bash can't handle structured data well. Python (being general purpose programming language) can't handle calling external programs well because it's not the "focus" of the language. My shameless plug solution is a "real" programming language that can do both well (along with proper handling of exit codes and more goodies for "devops"y scripting).
https://github.com/ngs-lang/ngs
More about this feeling when you don't have a well fitting language for "devops"y scripting - https://ilya-sher.org/2017/10/10/why-i-have-no-favorite-prog...
I personally use murex but Elvish seems popular too.
Not always no.
Even if your point were true, being programmer-first is not always a desirable feature. The vast majority of shell work is basic and repetitive. Most of the time people just want something that functions a little like Bash but less shitty for scripting. The fact that Powershell, LISP nor Python haven’t taken over the world for shell usage proves that a higher level language REPL generally makes for a shit daily shell. So usually you end up with just a small few purists using it. And that’s not really good enough.
Whereas Oil, Elvish, Murex and even Fish (to a less dramatic extent) are looking at what makes a good shell and then fixing shell scripting within that shell. Having gone through the REPL phase myself with a great many different languages and found them all painful for daily use, I’m inclined to agree with shell-first approach.
> Multiple dispatch would probably be the most prominent manifestation of this approach in NGS.
Again, NGS isn’t unique in that regard. Murex has methods, Powershell has methods. I’ve not seen anything new in NGSs “Multiple Dispatch” nor “MultiMethod” docs that other alt shells aren’t also doing.
Don’t take this the wrong way, it looks very impressive what you’re doing. But it’s not unique these days. And I say this having tried a great many options out there.
programming-language-first allows convenient scripting. I could not think of a way to make anything shell-first be convenient for scripting (anything beyond tiny scale).
> higher level language REPL generally makes for a shit daily shell.
I think it proves that general purpose languages, where using them as a shell was afterthought "makes for a shit daily shell".
NGS, on the other hand, has somewhat-bash-like (read: easily run external programs) syntax at the top level, which should be good for CLI. Want to use more advanced features that are typically associated with "real" programming languages? OK, pay the price, switch syntax with { ... } and have full blown programming language at your disposal, in the shell.
> NGS isn’t unique in that regard.
I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments?
> Murex has methods, Powershell has methods.
Just looked at Murex docs. No multiple dispatch mentioned. I did not even see how to define named parameters - https://murex.rocks/docs/commands/function.html
There is a huge difference in having methods and multiple dispatch.
> not unique these days
NGS is a mixture of "borrowed" features and unique ones. Multiple dispatch is an old concept and has been implemented in other programming languages. Examples of things in NGS that I have not seen anywhere else: syntax for run-command-and-parse-output, proper handling of exit codes.
Regarding exit codes. Typical approach to exit codes varies. Python - "I don't care, the programmer should handle it". Some Python libraries and other places - "Non-zero is an error". That's simplistic and does not reflect the reality in which some utilities return 1 for "false". bash (and probably other shells too) is unable to handle in a straightforward manner situation where external command can return exit codes for "true", "false" and "error". It just doesn't fit in the "if" with two branches. NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code.
Edit: and some other features around handling exit codes such as short syntax to provide expected exit code (any other exit code throws exception)
Edit: clarification - NGS knows that some external programs have exit code 1 which does not signify an error.
Murex does this but in a slightly different way. Rather than method overloading (which is a bad feature in any language in my opinion but I get it has its fans) murex has APIs that allow for writing data type agnostic methods. in practice that means the same tools to query a length of an array, or grab an item in a map (etc) work irrespective of whether the data type is a JSON array or map, YAML, CSV, S-Expressions or even just ‘ps’ (etc) output…and so on and so forth. This expands out to all methods, meaning you basically have a ‘jq’ like toolset that works exactly the same - same commands etc - for most data formats (certainly the ones I use daily). Which in effect is the same end result as what you’re describing except without the method overloading and instead allowing me as a shell script writer not to worry less about the data type or format of the piped data.
There are a few specific methods that do allow for overloading though, but that’s where functionality is a little more complex (these are usually event handlers though. I don’t use them much so I can’t say how well they work)
> NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code.
Murex does this too. I’m pretty sure I’ve seen other shells do that as well (possibly Elvish?)
Iconically it was actually the error handling that drew me to murex and the smart handling of data types within methods that kept me there. Basically all the same features you’re promoting in NGS. Which is why I’m saying there is a lot out there that does the same.
Anyhow, it’s been interesting reading about your work on NGS. Good luck
> I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments?
I've not heard of the term "multiple dispatch" but reading the thread it sounds like you're describing function overloading. Powershell does support this with classes[0].
murex does it's overloading at the API level[1]. The reason behind that decision is to keep the methods simple (eg a pipeline might contain JSON or CSV data) because you as a shell user don't want to run into situations where you've written a function that supports one data type but not another, you just want it to work first time. So murex automatically abstracts that part away for you. In addition to the point the other poster mentioned about consistent `jq`-like methods that are data type agnostic, murex allows for easy iteration through objects agnostic of their data type (eg `open somedata -> foreach { do stuff }` -- where you don't need to think about data types, file formats, etc, murex does the heavy lifting for you in a consistent and predictable way).
There are some specific builtins that support "overloading" of sorts but instead of overloading the function they have handlers that call user defined functions[2][3]. This removes the mystery behind function overloading because you can easily trace which handlers exist for which data types.
It's also worth adding that function overloading is supported in the same way that its also supported in Bash too where you might have a function but if there is a alias of the same name that will take priority. There are also private[4] functions (which are namespaced) so you have additional controls to avoid accidental overloading when writing modules too. And the interactive shell expands out any command (eg displays a hint text stating if a command is a function, alias, external command, etc) so there's a reduced risk of being unaware if `ls` is an alias, function, `/bin/ls`, or even a symlink to something else.
> I did not even see how to define named parameters - https://murex.rocks/docs/commands/function.html
Named parameters are optional because neither Windows nor POSIX have any understanding of named arguments in their processes. I did consider abstracting that into murex's functions regardless like Python et al would but I couldn't design a way that wasn't jarring nor cumbersome to write in a hurry nor worked transparently with Windows and POSIX ARGS[]. So I've come up with an optional builtin called `args`[5] which allows you to define flags, which are effectively the same thing as named parameters except they're supported natively by Windows and POSIX ARGS[], so you can write a murex script as a native shell script without leaving the user to write another abstraction layer themselves interpreting an indexed argument into an named arguments. Thus giving you the flexibility to write really quick Bash-like functions or more verbose scripting style functions depending on your need.
> Examples of things in NGS that I have not seen anywhere else: syntax for run-command-and-parse-output, proper handling of exit codes.
Both of these are baked into murex as well. The run-command-and-parse-output is the API stuff mentioned above. It's where the overloading happens.
As for error handling: any command is considered a failure if there is either a non-zero exit code, STDERR contains a failed message (eg "false", "failed", etc) or STDERR is > STDOUT[6]. This has covered every use case I've come across.
Additionally STDERR is highlighted red by default and when a process fails you're given a mini-stack trace showing where in the script the error happened. You have try/catch blocks, saner `if` syntax etc that all use the same API for detecting if a process has failed too. Keeping everything consistent.
The other thing murex has to help catch bugs and error is a testing framework baked into the shell language. The docs for test[7] need expanding but in essence:
- you can write proper unit tests
- you can intercept STDOUT (even if it's mid pipeline) and test that output. This works around instances where (1) you can't have full on unit tests due to side effects in a function that can't be easily mocked (2) you want to add your own debugging routines (rather than just printing values to STDOUT or running commands manually to see their output -- like one would normally have to do with shell scripting)
- you can add state watches. Except in murex the watches are Turing complete so you're not just adding noise to your terminal output but rather putting meaningful debug messages and scripts.
And all of the debugging stuff can be written straight into your normal shell routines and cause no additional execution overhead unless you've purposely enabled test mode.
So its fair to say I've spent a significant amount of time designing smarter ways of handling failures than your average shell scripting language. As I'm sure you have too.
> Regarding exit codes. Typical approach to exit codes varies. Python - "I don't care, the programmer should handle it". Some Python libraries and other places - "Non-zero is an error". That's simplistic and does not reflect the reality in which some utilities return 1 for "false". bash (and probably other shells too) is unable to handle in a straightforward manner situation where external command can return exit codes for "true", "false" and "error". It just doesn't fit in the "if" with two branches. NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code.
If you're describing that as a "two branch" approach then technically murex could be argued as having "three branches" because it checks exit code, STDERR contents, and payload size too. Personally I just describe it as "error handling" because in my view this is how peoples expectations are rather than the reality of handling forked executables.
I guess where NGS and murex really differ is NGS likes to expose its smart features whereas in murex they're abstracted away a little (they can still be altered, customised, etc) to keep the daily mundane shell usage as KISS (keep it simple stupid) as possible. eg you can overload function calls if you really wanted but that can often cause unforeseen complications or other unexpected annoyances right when you least want it to. So murex keeps that stuff around for when you need it but finds ways to avoid people needing to rely on it and furthermore unwraps the covers of what a routine does in the interactive terminal. It's all about offering abstractions but removing surprises.
> NGS knows that some external programs have exit code 1 which does not signify an error.
Having different behaviours for different executables hard coded into the shell is one behaviour I purposely avoided. I do completely understand the incentive behind wanting to do this and wouldn't criticise others for doing that but given external programs can change without the shell being aware, they can be overloaded with aliases, functions, etc, and they might even just differ between Linux, BSD, Windows, etc -- well it just seemed like hard coding executable behaviour causes more potential issues than it solves. You also then run into problems where users expect the shell to understand all external executables but there are some you haven't anticipated thus breaking expectation / assumptions. Ultimately it creates a kind of special magic that is unpredictable and outside the control of the shell itself. So instead I've relied on having foundational logic that is consistent. It's the one side of shell programming where I've placed the responsibility onto the developer to get right rather than "automagically" doing what I think they are expecting. This of course does mean I have to focus even harder on ensuring all other aspects of the shell are predictable and low maintenance so that the developer can cover any specific edge cases with ease. Which goes a long way to explaining why I've chosen the path I've chosen (ie reducing the amount of syntax sugar, overloading, etc needed for daily use that one might rely on in a more traditional programming language).
Links:
[0] https://stackoverflow.com/a/57294942
[1] https://murex.rocks/docs/GUIDE.apis.html
[2] https://murex.rocks/docs/commands/openagent.html && https://murex.rocks/docs/commands/open.html
[3] https://murex.rocks/docs/commands/event.html (I need to write a lot more documentation around event driving programming in murex)
[4] https://murex.rocks/docs/commands/private.html
[5] https://murex.rocks/docs/commands/args.html
[6] https://github.com/lmorg/murex/blob/19d2aeb27e814a4c6fec77d2...
> I've not heard of the term "multiple dispatch"
https://en.wikipedia.org/wiki/Multiple_dispatch (includes example from NGS too)
In NGS, multiple dispatch is based on CLOS model described at https://en.wikipedia.org/wiki/Common_Lisp_Object_System , with my own simplifications.
> Powershell does support this with classes
Somewhat similar. Since methods do not live in classes in NGS, I would argue the mechanism in NGS is simpler (more elegant?). You can just define your_method(c1:Class1, c2:Class2).
> murex does it's overloading at the API level[1].
mmm. I see the support but looking at documentation at https://murex.rocks/docs/apis/Unmarshal.html , I see it's not exposed into the Murex language, it's in Go. Is this correct?
> where you don't need to think about data types, file formats, etc, murex does the heavy lifting for you in a consistent and predictable way
That is good and what I would expect.
( sorry, running out of time, to be continued :) )
Learn something new every day. Seen (and used) this methodology before but wasn't aware it was called that. I'd always heard of it as "overloading" which is conceptually similar but not identically the same.
> mmm. I see the support but looking at documentation at https://murex.rocks/docs/apis/Unmarshal.html , I see it's not exposed into the Murex language, it's in Go. Is this correct?
It's APIs written in Go (for performance and convenience -- you wouldn't want to write a YAML marshaller in a shell scripting language). So the methods are exposed as builtins. However you can add method written in murex if you wanted.
The idea being get 99.9% right by default but leave some flexibility for the user to customise if they want.
Murex builtins are also written in a way that they're all optional includes into the core project. Thus you can easily write your own builtins, marshallers, etc in Go if you want performance and then call them from your shell script (or you could write them in Python, Java, Perl, etc and run them as an external executable if you wanted too. But if you do that you lose access to murex's typed pipelines. Which is where the really interesting stuff happens (I haven't yet figured out a non-shitty way to send typed data over POSIX pipes to external executables).
> sorry, running out of time, to be continued :)
Any time :) I'm finding this conversation really interesting and educational
In NGS the common case for multiple dispatch would be to define your own type and then add methods to existing multimethod to handle that type; additionally I've simplified the dispatch algorithm exactly for this reason - unclear answer to "what's going to be called actually?".
> function overloading is supported in the same way that its also supported in Bash too where you might have a function but if there is a alias of the same name that will take priority.
I don't think it is called overloading. That's very different from having two methods with the same name and the "right one" is called based on types of passed arguments.
> interpreting an indexed argument into an named arguments.
In NGS it happens in exactly one place, when main() is called. At that point command line arguments are automatically parsed based on main() parameters and passed to main().
I don't think I understand your reasoning behind not including named parameters.
> NGS likes to expose its smart features whereas in murex they're abstracted away a little
Sounds about right. Power to the people!
> Having different behaviours for different executables hard coded into the shell is one behaviour I purposely avoided.
I can see why. NGS prefers to do the right thing in most cases and have shorter and cleaner code. Yes adding some risk, which I see as not that big - exception when an exit code was "ok" for unknown program (unknown external programs default to exception on non-zero exit code).
I also dream about moving the hard coded information about programs into separate schema, which could be re-used between different shells. That would be similar to externally provided typescript definitions for existing code, described at https://www.typescriptlang.org/docs/handbook/declaration-fil...
> differ between Linux, BSD, Windows,
"detect program variant" feature in the schema I mentioned above
> hard coding executable behaviour
Yep, smells a bit
> causes more potential issues than it solves.
I think in particular situation with exit codes the risk is low.
> users expect the shell to understand all external executables but there are some you haven't anticipated thus breaking expectation / assumptions
Yes. Downside. That's the kind of surprises that suck. Need to make sure at least the docs are clearly warning about this.
> So instead I've relied on having foundational logic that is consistent.
Totally understandable.
> STDERR contains a failed message (eg "false", "failed", etc) or STDERR is > STDOUT[6]. This has covered every use case I've come across.
Sounds also like potentially surprising, somewhat similar to handling exit codes. Yes, it's not per program and that's why it's better but still.
> Additionally STDERR is highlighted red by default
As it should! Yes!
> mini-stack trace
Yes, please!
> testing framework baked into the shell language.
Same in NGS but https://github.com/ngs-lang/ngs/issues/494 :)
> So its fair to say I've spent a significant amount of time designing smarter ways of handling failures than your average shell scripting language. As I'm sure you have too.
I guess. Differently of course :)
> Personally I just describe it as "error handling" because in my view this is how peoples expectations are rather than the reality of handling forked executables.
Sounds like I was not clear enough as if something is not uniform in this regard in NGS. Exceptions are not only for external programs. It just happens that external program can be called inside "if" condition; an exception might occur there.
> I don't think it is called overloading. That's very different from having two methods with the same name and the "right one" is called based on types of passed arguments.
Yeah you're right that's not overloading. I'm not really sure why included that in my description.
> In NGS it happens in exactly one place, when main() is called. At that point command line arguments are automatically parsed based on main() parameters and passed to main(). > I don't think I understand your reasoning behind not including named parameters.
there isn't really an entry point in murex scripts so
function foobar {
out foobar
}
foobar
#!/usr/bin/env murex
out foobar
$ out foobar
I wasn't really interested in creating a two tier language where parameters are conceptually different depending on what's being called so I decided to make flags easy instead:
function hippo {
# Hungry hippo example function demonstrating the `args` builtin
args: args {
"Flags": {
"--name": "str",
"--hungry": "bool"
}
}
$args[Flags] -> set flags
if { $flags[--hungry] } then {
out: "$flags[--name] is hungry"
} else {
out: "$flags[--name] is not hungry"
}
}
- `args` still contains more boilerplate code than I'm happy with
- murex cannot use `args` to automatically generate an autocomplete suggestion
So I expect this design will change again.
> Sounds also like potentially surprising, somewhat similar to handling exit codes. Yes, it's not per program and that's why it's better but still.
I don't see how that's surprising because it's literally the same thing you were describing with NGS (if I understood you correctly).
> Same in NGS but https://github.com/ngs-lang/ngs/issues/494 :)
lol I like it :)
> Sounds like I was not clear enough as if something is not uniform in this regard in NGS. Exceptions are not only for external programs. It just happens that external program can be called inside "if" condition; an exception might occur there.
It was me who was unclear as I assumed you meant it was the same error handling for builtins and functions too (as is also the case with murex). I just meant the "reality of handling forked executables" as an example rather than as a commentary about the scope of your error handling :)
I have a nice trick in NGS for that. Under the idea that "small scripts should not suffer", script is running top to bottom without "entry point". However, if the script has defined main() function, it is invoked (with command line arguments passed).
Example - https://github.com/ngs-lang/nsd/blob/afe0cad5e506ec4ee2fa924...
> `args` still contains more boilerplate code than I'm happy with
Is there anything preventing you to have exactly the same functionality but with syntactic sugar that it looks like parameters declaration? (Just to be clear, keeping all the ARGV machinery).
Something like (assuming local variables are supported; if not, it could still be $args[Flags] etc):
function hippo(name:str, hungry:bool) {
if { $hungry } then {
out: "$name is hungry"
} else {
out: "$name is not hungry"
}
}
Yes it is almost the same in NGS with regards to exit codes, which you preferred not to do in Murex. On the other hand, checking stderr looks very similar to knowing about exit codes and here you decided to go for it. I'm puzzled why. It is somewhat fragile, like knowing about exit codes.
> It was me who was unclear as I assumed you meant it was the same error handling for builtins and functions too (as is also the case with murex). I just meant the "reality of handling forked executables" as an example rather than as a commentary about the scope of your error handling :)
Here I lost you completely but I hope it's fine :)
In NGS, exception handling is used throughout the language, consistently, well at least I hope it is.
From what I understood - same in Murex.
> Something like (assuming local variables are supported; if not, it could still be $args[Flags] etc):
> example code
oooh I like that idea. Thank you for the suggestion.
> Yes it is almost the same in NGS with regards to exit codes, which you preferred not to do in Murex. On the other hand, checking stderr looks very similar to knowing about exit codes and here you decided to go for it. I'm puzzled why. It is somewhat fragile, like knowing about exit codes.
No, murex still checks for exit codes. Essentially a program is considered successful unless the following conditions are met:
+ exit code is > 0
+ STDERR in ('false', 'no', 'off', 'fail', 'failed')
+ or []byte(STDERR) > []byte(STDOUT)
STDERR messages are useful for functions that return a message rather than exit code. Particularly inside `if` blocks, eg
= 1 == 2 -> set math
if { $math } then { out: $math } else { err: $math }
As for the STDERR > STDOUT test, that covers an edge case where utilities don't return non-zero exit codes but do spam STDERR when there are problems. It's an uncommon edge case but the only time this condition is checked is when a command is evaluated in a way where you wouldn't normally want the STDOUT nor STDERR to be written to the terminal.
This also allows you to get clever and do stuff like
if { which: foobar } else { err: "foobar is not installed }
As in not throwing exception or as in evaluates to true?
> exit code is > 0
You refused to get into specific exit codes for specific programs citing potentially surprising behavior (which I agree will sometimes be an issue).
> STDERR in ('false', 'no', 'off', 'fail', 'failed')
I correlated this with the following source code:
> if len(s) == 0 || s == "null" || s == "0" || s == "false" || s == "no" || s == "off" || s == "fail" || s == "failed" || s == "disabled" { return false
... which also has a potential for surprising behavior. (side note: appears to work on stdout, not stderr)
> or []byte(STDERR) > []byte(STDOUT)
Does that mean len(stderr) > len(stdout) ?
... also has a potential for surprising behavior. I can easily imagine a program with lots of debug/warnings/log on stderr and truthy/non-error output.
> STDERR messages are useful for functions that return a message rather than exit code.
Might be problematic because typically exit codes are used for that.
> where `if` is evaluating the result of `which` and you don't need to do more complex tests like you would in Bash to test the output of `which`.
Mmm.. Never did this. In which situation looking just at exit code of `which` is not good enough?
> if { which: foobar } else { err: "foobar is not installed }
Just to show off :) in NGS that would be:
if not(Program('foobar')) {
error("foobar is not installed")
}
assert(Program('foobar'))Perl and Ruby do both very well. Or at least they do the latter in a simpler way than Python, and they're no worse than it for the former.
1. Tab completion. All major bash-completion packages know how to show Makefile targets in response to a TAB.
2. Parallel, serial, or single-target execution.
3. Automatic dependency resolution between tasks. Tasks that build files can also use timestamps to see what needs rebuilding.
4. Discoverability. Anybody who sees a Makefile will usually understand that something is supposed to be run from the Makefile's directory. Chances are good that they'll check the tab-completions too. There are conventions for standard targets like 'clean' and 'all'.
If you have a project with a build-process that has a bunch of small tasks that you might sometimes want to run piece-by-piece, Make is the perfect tool IMO.
Learning a handful of bash idioms and best-practices has made a massive impact for me, and life much easier. The shell is something you cannot avoid if you're a programmer or other sort of code-wrangler.
You can interact with it + be (mostly) clueless and still get things done, but it's a huge return-on-investment to set up "shellcheck" and lookup "bash'isms", etc.
----
(Off-topic: I am convinced Ruby cannot be beaten for shell-scripting purposes. If I had a wish, it would be that every machine had a tiny Ruby interpreter on it so I could just use Ruby. I'm not even "a Ruby guy", it's just unreasonably good/easy for this sort of thing. And I keep my mind open for better alternatives constantly.)
Example of near-identical script in bash vs Ruby:
https://github.com/GavinRay97/hasura-ci-cd-action/blob/maste...
https://github.com/GavinRay97/hasura-ci-cd-action/blob/maste...
I'm not sure how much closer to describing your exact intent in English a language can get than:
successfully_made_executable = system 'chmod +x /usr/local/bin/hasura'
abort 'Failed making CLI executable' unless successfully_made_executable
EDIT: Looks like Perl/Raku is essentially the same as Ruby in this regard. So besides it being a whacky language, take that for what you will:
$successfully_made_executable = shell "chmod +x /usr/local/bin/hasura"
die 'Failed making CLI executable' unless $successfully_made_executable.exitcode is 1> successfully_made_executable = system 'chmod +x /usr/local/bin/hasura'
> abort 'Failed making CLI executable' unless successfully_made_executable
I guess, but if I was writing scripts like this, I'd really want to write a helper method something like:
def system_or(cmd, fail_msg)
system cmd || abort fail_msg
end
# and then
system_or 'chmod +x /usr/local/bin/hasura', 'Failed making CLI executable'
I wrote this a few years back which ran on bunches of hosts and fed into an infrastructure network mapper based on each hosts' open network sockets to other known hosts. It wasn't really feasible to install a set of tools on random hosts.. but I still had root ssh access across the board. So I needed something tool agnostic, short, auditable, and effectively guaranteed to work:
What does "export" mean in this context? Do you just mean `ssh host cmd` where cmd is some shell that happens to contain (and call) a function?
Try this and you'll see how it returns a dramatic amount of bash as its output:
remote_cmd="`typeset -f` ; function test123 { echo hi! ; }" && echo -e $remote_cmdI would imagine there would be performance implications of defining every bash function as a subshell, is why it's not universally recommended to define functions this way?
No?
#!/bin/bash
fib1() {
if (( $1 < 2 )) ; then
echo $1
return 0
fi
echo $(( $(fib1 $(($1-1))) + $(fib1 $(($1-2))) ))
return 0
}
fib2() (
if (( $1 < 2 )) ; then
echo $1
return 0
fi
echo $(( $(fib2 $(($1-1))) + $(fib2 $(($1-2))) ))
return 0
)
time fib1 15
time fib2 15
610
real 0m1.585s
user 0m0.590s
sys 0m0.972s
610
real 0m3.168s
user 0m1.068s
sys 0m2.025sCreating a new process for every single function invocation seems crazy to me, and but might actually be just fine for many "ordinary" use cases? (Although might not have been on computers of 20+ years ago, which might also be why it's not something advised, so much of bash tradition is decades old?)
Actually Oil functions don't use dynamic scope, but this is done in-process, not with another process:
https://www.oilshell.org/release/latest/doc/variables.html#p...
----
Also nested functions don't really add much useful to shell. It's purely a matter of textual code organization and doesn't affect the semantics. I define all functions at the top level.
So arguing about the cost of sub-shells is somewhat besides the point.
Are there really cases where subshells are invoked within the same process? In my experience, it has never been the case. That's why I've been trying to minimize the use of subshells because spawning a new process is a bit slow.
ARG_ARCH="amd64"
ARGUMENTS+=(
"ARG_ARCH,a,arch,Target operating system architecture (${ARG_ARCH})"
"log=utile_log,V,verbose,Log messages while processing"
)
utile_log() { echo "$1" }
noop() { return 1 }
log=noop
preprocess() {
$log "Preprocess"
}
https://dave.autonoma.ca/blog/2019/05/22/typesetting-markdow...
Here's a rudimentary implementation and example usage to wet your whistle:
https://github.com/DaveJarvis/keenwrite/blob/master/scripts/...
https://github.com/DaveJarvis/keenwrite/blob/master/installe...
run_foo() { echo foo; }
run_bar() { echo bar; }
name=foo
run_$name> "what if that variable isn't set to whatever the developer is expecting it to be set? What random command will then be executed?"
As an author of an alternative shell myself, I purposely removed this ability for that reason.
While I do agree that it would be nice to be able to have 'local' functions and have inter-function cleanup work better, the logical conclusion for me was not to use function subshells. Since the use case is for larger programs (where different functions may want to have their own cleanup mechanisms), I'm opting to go for more of a library route. For example, I'm working on a Bash library that includes a function to allow different sources to add (and remove) functions to the same `TRAP`. A similar function may be useful, possibly involving the `RETURN` trap and the `-T` flag for the use case the author brings up. Obviously, using a package manager for _Bash_ of all languages brings in a lot of overhead, but I think it can be quite powerful, especially with a potential "Bundle" feature that makes scripts work without the package manager.
Concerning specifically the use of subshells, (as other commenters have pointed out) it significantly reduces performance. I also disagree that dynamic scoping is necessarily bad for Bash. I find it quite useful when I need to use various common functions to manipulate a variable - since modifying and 'returning' variables from a function is usually either slow or verbose with Bash. Admittedly though, this feature is quite annoying at times - for example, most public functions in my Bash package manager[2] all have their variables prefixed with two underscores - because they `source` all the shell scripts of all package dependencies - so I want to be extra certain nothing weird happens
[1] https://github.com/hyperupcall/bash-core/blob/a17ab0a8b6070f...
[2] https://github.com/hyperupcall/basalt/blob/main/pkg/lib/publ...
However, I would strongly advice to master a proper programming language. I respect the article and the efforts of the author, but I feel that it is the past.
I mastered Python a bit and the ability to just use things like dictionaries, proper parsing libraries and such, instead of kilometers of fragile pipes, it is so much better.
I understand something like Python may feel total overkill, but that 10 line shell script suddenly needs quite a bit of error handling and some other features and before you know it, you wish you started out with python or something similar.
Besides cleanup, one thing I think I would love to see is a good mechanism for logging. I have started to build a file for functions and then other files, which source that as a library and calls the functions as needed. I would love to be able to tell the library functions to log something if the parent file wants it, print to stderr or stdout by default or be silent if the caller wants that instead.
No, it wasn't smart to do so.
Yes, it still works.
No, I don't know how it works.
{
foo
} always {
cleanup stuff
}`local` is not in POSIX, but is so common in "posix" shells that many use it, just a sidenote.
So maybe it is about bash functions and not posix functions, because it's refencing features that exist in bash but not posix?
https://www.anardil.net/2018/dynamic-shell-scripting.html (self plug)
I recently discovered, similar to the author of the post for this thread, that local variables are dynamically scoped.
I have been writing a lot more shell scripts lately, using a "library" [1] of sorts I've been writing. When I was debugging one of my scripts that uses mycmd, I discovered that I had failed to declare some of my variables local and they were leaking out to the global scope.
I had recently added functionality to call a set of functions on script exit, so I added something that would output the defined variables, in hopes that I could write something that will output them at the beginning and then the end and show the difference. I was surprised when variables defined in my dispatch function [2] for those at exit functions were showing up, even though they were definitely defined as local. It was then that I dug around and discovered the dynamic scope of variables.
I've been trying to figure out how to accomplish what I desire but exclude those variables from calling functions. I haven't been able to find an obvious way to see if the variable is coming from a calling function. I might be able to use techniques like you've pointed out in your linked post to add the tracing that I want. Still need to think more on this.
---
[1] https://github.com/travisbhartwell/mycmd [2] https://github.com/travisbhartwell/mycmd/blob/main/shell/myc...
Is there a way to automate this as a shell script without having to type the password?
I assume
Sudo -s | password | unbind string but that doesn't work.
The other day I just made a sample stripe-jobs cli app with it and playwright.
foo () ( x="hello"; echo $x )
Or anything that can handle integers and floating point calculations natively.
I think pretty much the only reason people write bash is because you have an incredibly high chance of bash being installed and a version of bash being installed that will run whatever bash you write just fine.
Perl is honestly almost as reliable to be there predictably and compatibly... but I guess people would rather write bash than Perl?
I feel like what usually makes me reach for something beyond Bash is really a matter of wanting or needing some dependency that wasn't written in it for whatever reason. Usually this happens right at the point where the script/utility starts to turn into a library/program, so it's trivial to just transpose the control flow into whatever language is required at that point and go from there. This of course raises the type of concern you mentioned about Ruby, but at that point it's hopefully worth the trouble to address.
And yes, you have these sorts of problems with other languages, but my point here is that Bash doesn't free you from them.
Or you can just use bash and keep your sanity.
This justification for bash has always been perplexing to me. If I’m operating in an environment where my ONLY reliable infra invariant is “bash will probably work”, cleaning up the org’s infrastructure clusterfuck is probably my #1 priority.
(Or I guess in a few cases the fleet are not boxen but little embedded/iot devices, in which case you probably don't want to be running any of these sorts of scripts for a whole host of reasons...)
> of course you shouldn’t install random gems globally… Or you can just use bash and keep your sanity.
What are some scenarios where you’d need a gem in Ruby but Bash just works?
The only situations I can think of are cases where you’re using Bash to call out to some executable that does fancy things. Which (1) you can do from any scripting language anyways, and (2) means you’re just shifting dependency management from the package manager to the image/Dockerfile.
But actually, the combination of justifications here is particularly perplexing. If the org has a stable way of handling system images, then you'll know which version of $scripting_language should exist and where it's installed. The only way you end up with language version woes is if you don't have standardized infra. BUT... if you don't have standardized infra, and Bash can do things that Ruby can't without special Gems, then it stands to reason that you're in a situation where your Bash scripts depend on the magic state of individual unicorn boxes?! Which is particularly fragile and frightening and far worse than installing some local gems or whatever!
IDK. The purported benefits of Bash always sound like they flow out of environments where there are basically no fleet-wide infrastructure invariants.
"Just use $scripting_language" might be the best advice in this thread just as a sort of canary in the coalmine. I.e., if your org can't "Just use $scripting_language" because "which version?" then the team will probably benefit tremendously in an infinite variety of ways from an afternoon of infrastructure cleanup. Regardless of whether they use bash or a scripting language going forward :)
Also, don’t even start me up on self-modifying code, we have one at a work project and it sometimes just fails and results in inserting the same echo statement at each run, resulting in every bootstrap displaying 2^n messages, depending on when have I last cleaned it up…
Undeniably, yes. I was there in the 90s ;-)
But hacking things together in a way that's robust is difficult, and bash isn't a good match for that difficulty.
These days I mostly operate in the realm of "how can I enable others to hack things together without blowing tens of millions of our dollars and their very early career on a stupid mistake".
That is not your job. Your job is to get that machine working using whatever is already installed. Adding a new package means going to the production committee with your proposal and justification and analysis of the increased threat surface.
The defaults are everything.
> That is not your job.
This sort of thing is definitely your job have the word "Principal" in your job title, and probably also if the word "Senior is in there as well ;-)
And in any case everyone is responsible for excellence in the milieu in which their team operates. If Senior or even fresh grad Jr. comes to me with a solid good idea I'll champion for it as if it were my own baby. And then recommend/fight for rapid promotion in the case of Jr or put in a good word for promotions for the Sr.
If you recommend your org have standardized images with well-documented info about language versions etc. and the answer you get from your management/tech leadership is "not your job", I recommend finding a new job.
> Adding a new package means going to the production committee with your proposal and justification and analysis of the increased threat surface.
The context of my quote was "knowing which version of ruby/perl/python is installed". There's almost certainly a version of one of those on your standard linux machine, and everyone pushing to prod should damn well be able to look up exactly which one.
> Adding a new package
The general debate here goes way beyond adding a new package. Good infra needs WAY better invariants than "definitely bash is installed in the usual place". If a concern is "IDK which version of Ruby is installed on the machines I'm targeting" then either you're fighting fires and need to keep every intervention really damn simple or else your org as Real Issues. In either case, bash is the enemy.
> The defaults are everything.
Those defaults aren't handed down from Gods. Your org chooses them.
Despite this you can expect your bash scripts written today to work on any of these machines and their ancient OS installs. This is primarily because people writing in Bash care enough to not use new features, not the lack of new features in Bash.
Additionally, there are tons of devices out there that will never get updated that run even older unsupported linux distros and do their job just fine. And modern written bash will run just fine there too.
I am not holding my breath.