Hacker News Reader

Comment by throwaway984393 | Hacker News Reader

Getting anything out of a subshell that isn't from STDOUT is impossible. So you can't define an array in a subshell and then use it outside the subshell, and you can't return an array (or anything that isn't a string) from a subshell. If you only use subshells and want to use any kind of data structure that isn't a string passed from STDOUT, you have to do it globally. And subshells are slow. So nobody uses subshells.

If you use Bash for programming, you have to stop thinking in terms of the holier-than-thou software engineer, whose ego believes that a superior, "clean" design makes a superior program. You should embrace globals. You should switch between using or not using the enforcement of set variables or program exit status. You should stop using Bashisms and subtle, obscure language features unless you absolutely have to.

Bash is not a "real" programming language, so do not treat it as one. Do not look for hidden features, or try to do things in cute ways that nobody else uses. There is no superior method or hidden knowledge. Just write extremely simple code and understand the quirks of the shell.

View on HN

This reminds me of the arguments made against writing “real” code in JavaScript in the early days of the web, until Crockford came along and wrote “The Good Parts.” There is no reason to think that a few idioms and curating features could go a long way to leading to a much better, less hacky paradigm for bash shell scripting.

Bash is much older than JavaScript. If it was going to turn into a real programming language, it would have by now. It hasn't.

Also, JavaScript really is not a very good programming language. We are just stuck with it because it's the only language the browser understands. (Well, until recently: things are going to change with the introduction of wasm).

But for the shell, we're not stuck with any one language. Whatever you want to do can be programmed in your favorite language. You can easily write a python script instead of a bash function.

> things are going to change with the introduction of wasm

people have been saying this for years already and nothing seems to have come of it :(

I've seen some pretty interesting things like the use of Rust for front end development, like yew[1] and Seed[2].

There aren't many languages that are practical for WASM output. Scripting and managed languages need to ship their interpreters and runtimes with their WASM blobs, and can end up relatively large. JavaScript's interpreter and runtime are baked into every browser already.

That leaves only compiled and unmanaged languages for potentially good WASM targets. As mentioned before, Rust is seeing a lot development in that space. If LLVM can compile it, then Emscripten can output it to WASM.

[1] https://github.com/yewstack/yew

[2] https://github.com/seed-rs/seed

Are you sure nothing came of it?

Figma for example seems to be largely powered by WASM and the core of the product seems to be written in C++

https://www.figma.com/blog/webassembly-cut-figmas-load-time-...

https://www.figma.com/blog/building-a-professional-design-to...

I think the majority of web developers are just slow at catching up with this new things.

Is that true though? Aren’t people using it for in-browser image and video editing? Doesn’t Google use it for drawing to Sheets and crunching numbers?

I thought that was the case but I can’t recall where I read it. If that’s not true then I should probably reconsider my thinking about wasm

Yes, WASM is a good target for CPU bound number crunching. I'm not sure if it's used in Sheets, though.

Wasm isn't really going to change the front-end / browser side of javascript for the better, but it's great for graphics and anything else cpu intensive.

WASM DOM manipulation will probably come soon, and that will place other languages in the exact same playing field as Javascript. For now, manipulating the DOM still requires calling Javascript.

Yeah. Turns out, having an interpreted runtime in the browser is quite useful and sending bytecode across the wire not as practical.

We've been forgetting and relearning that for the past two decades.

"turns out"? How did it turn out so?

Actually it turns out that having an interpreted runtime causes a stream of endless problem. We keep try to patch them over but new problems keep coming up.

It turns out that a standarized low level byte code is the right way to ship cross platform applications. We already knew that. It just took a long time to go through the standarization process and be implemented by browsers.

If it seems like it's not in use today, it's largely because JavaScript has momentum and the majority of web programmers don't yet know how to take advantage of wasm (or maybe don't think they need it).

> It turns out that a standarized low level byte code is the right way to ship cross platform applications

So why hasn’t it taken over the world with Java and now with wasm?

> If it seems like it's not in use today, it's largely because JavaScript has momentum and the majority of web programmers don't yet know how to take advantage of wasm (or maybe don't think they need it).

There’s another possible reason. I’m sure you can come up with the answer yourself.

Javascript never really had a chance to "take over the world" before Google released Chrome and the V8 engine. Before that it was just too slow.

Java applets really sucked. They took a long time to load and initialize. They all looked ugly (probably because of the default libraries?).

Also Java itself as a language was really bad and the development experience was awful. I want to say that no one wants to program in Java but in reality many people do (I don't understand those people).

The important lesson here is the implementation is more important than the idea.

Good idea with bad implementation -> goes no where.

wasm is not java.

The important quality about wasm is that it's not garbage collected. It's pretty close to just good old assembly.

>Javascript never really had a chance to "take over the world" before Google released Chrome and the V8 engine.

Chrome was released at the end of 2008, JavaScript was thriving way before that. We had Gmail since 2004, jQuery since 2006. WebApps were the “sweet solution” considered for iPhone apps at first in 2007. Chrome exists because of the healthy ecosystem that Firefox and Safari provided, not the other way around.

>It's pretty close to just good old assembly.

Precisely, and it's yet to be proven that's the best solution for the Web. I love it from the computer science perpective, but historically that idea hasn't struck a chord.

  > JavaScript really is not a very good programming language.

Since 2015, JavaScript really is a nice language. I never thought that I'd say that.

Go look at ES6 (the javascript version that came out in 2015) and even TypeScript. You'll be pleasantly surprised, no matter if you're coming from C++, Java, C# or even other "scripting" languages such as PHP and Python.

I use typescript almost daily. It's certainly better than Python or Ruby, I'll grant you that. But I don't think Python is a good language either.

Ultimately I want a langauge with value types like structs and arrays that you can use to do computations without allocating things on the heap.

In Go I can write a function that returns two numbers.

In Javascript I can only return two numbers by allocating an array on the heap with two items. These two items themselves are probably pointers to two number objects allocated separately. So that's 5 allocations to return two numbers. It's insane.

This means fundamentally it's impossible to create applications in javascript that are both sophisticated and high performance.

A great example to illustrate this is the tooling around Javascript itself.

There were all kinds of bundlers: webpack, rollup, parcel, etc. Rollup was considered the fastest and lightest. They were all programmed in Javascript itself. They were all slow. But no one really knew how bad it was because there was nothing really better.

Then esbuild comes along, and blows all of them completely out of the water, out performing them by 100 times. And it's written in Go. A language that supports structs and arrays as value types.

It’s not fundamentally impossible to build fast apps in JS. It’s just not possible to do if you don’t do the legwork of writing code that the compiler will optimize for you. It’s not easy, and it’s not natural in a lot of cases, but saying it is fundamentally impossible goes too far: people build high performance applications like games in JS.

For example, you are worried about value types but under the hood the JIT compiler will actually generate efficient representations that are passed by value if you do not mess things up for it. Modern JS compilers are extremely sophisticated.

For returning values without an allocation, you’d create such a “struct” and then use a memory pool. It would still be heap allocated I believe but not totally sure. People who know more than I do could probably tell you a better method to return multiple values from a function efficiently.

> people build high performance applications like games in JS

Yes, and in my experience this kind of applications (and the browser in general) are one of the only things these days that still push me to upgrade my hardware to have better CPUs and more RAM, as it's perfectly sufficient as is for pretty much everything else I do.

You can’t tell from the outside if you are running a well engineered JS app that would have been fast in another platform, or a non-optimized JS app. My point was that it’s not fundamentally impossible to write fast enough JS code for a game, even though you may come across many slow JS apps regularly. (Not surprising, given the wide distribution and low barrier to entry of the web.)

Yes, you can operate strictly on typed arrays and essentially implement something alike a virtual machine inside your JS code to make it handle memory efficiently, so you're right - it's not "fundamentally impossible", but are you still really writing JavaScript at that point? :P

What’s wrong with modern JavaScript as a programming language (although I would maybe point to typescript which adds types to JS instead)?

There's no distinct "modern JavaScript", it's still the same language. It just has bunch of things added to make it more manageable, but it still contains all the footguns and gotchas as it did before.

Its tooling got significantly better in the last decade or so, but you still need to intentionally rely on it in order to work with the language in any sensible way. You can't really hope for any other outcome when maintaining backwards compatibility.

The tooling only got better last year with esbuild.

Everything else is either unmanageably complicated (webpack) or slow (rollup, parcel, babel).

React mainstreamed the idea of using a virtualdom to accelerate UI building (accelerate in the computational sense: by not dealing directly with the slow DOM).

I remember how webdev looked like 15 years ago. The tooling did get better even before esbuild - but I never said it was "good" :)

I responded to the sibling comment asking the same question

https://news.ycombinator.com/item?id=29068066

> There is no reason to think that a few idioms and curating features could go a long way to leading to a much better, less hacky paradigm for bash shell scripting.

A couple of years ago, I challenged myself to write some complex scripts and apps using only Bash, and came to this conclusion myself.

You can get pretty far with just Bash alone, especially if you strive to write readable and maintainable code, and not just one off scripts. If you need access to data structures, especially nested ones, you can shell out to another language and then print the results back to stdout so Bash can use them:

    $ get_data | transform_data | python3 ./complex_script.py | consume_results

But once you shell out and require another language, why not just use that other one from the start?

Because the shell is used to do things that are easy in the shell. Examples: file manipulation, sorting, field based processing, pipes. Doing these things in a traditional language is more complex and not necessarily better.

Except the shell is bad for those things, except pipes I guess.

I just hate doing things with files in the shell, all those stupid escaping rules, and god forbid you have files with spaces or quotes in the names, or leading or trailing spaces.

It's just spending time learning hacks to get around shell limitations instead of actually getting things done, and at the end it's an ugly mess.

Or maybe I'm just bad at it, but I don't think it's only that.

It’s not more complex in Python if you use the sh package. All the power of everything you need with a real language when you need to do more.

Packaging in Python is a bit more of a pain.

That's the other part of the problem. The shell is available everywhere. Python needs to be installed. So, if you need shell-related functionality, it is the easiest route to use.

I have not encountered a system yet where Python isn’t installed out of the box.

Windows, possibly. And as for the *nixes, including Macs, what version is installed? The way Python 2 eol was extended and extended and extended just hurt the whole ecosystem.

Windows doesn't have bash preinstalled so I'm not sure why you're bringing that up. On all OSes I've used in the past 3-4 years all of them have been Python3 by default. The specific version is largely irrelevant as I could ask you the same question about which version of bash is running on a given OS (e.g. it's not available on Mac out of the box as the default shell is zsh due to GPLv3).

  > Windows doesn't have bash preinstalled

I meant Python preinstalled.

  > On all OSes I've used in the past 3-4 years all of them have been Python3 by default.

Windows comes with Python preinstalled now?

I did a similar thing a couple of months ago, writing a static site generator for my blog in sh. You can write bigger programs as long as you keep discipline, albeit it is a pain.

Here is the rust library with the “the only good parts” of bash/shell script: https://github.com/rust-shell-script/rust_cmd_lib

"no reason to think ... could"

should read

"no reason to think ... couldn't"

> Getting anything out of a subshell that isn't from STDOUT is impossible.

You can use I/O redirection with arbitrary file descriptor numbers, and you can let the shell pick them. So you can `mknod` a pipe, `exec {pipefd_rw}<>"$pipe_name" {pipefd_ro}<"$pipe_name" {pipefd_rw}>"$pipe_name";` and now you have three open file descriptors that you can let the sub-shell use to communicate with the parent. And so you're not limited to stdin/stdout. I do wish there was a built-in for making an anonymous pipe instead of having to make one on filesystem.

(You have to open the named pipe for read-write first because otherwise you'll block, but if you don't want a read-write FD for it you can close it after opening the read and write FDs.)

> So nobody uses subshells.

I'll count myself among nobodies, I guess. I use them a fair bit when part of said subshell cd's somewhere, and I don't feel like using pushd/popd, either of which might fail to operate based on my mistakes, but subshells seem to never fail to exit, eventually.

Depends what you're doing. If you need the exit status of something _in_ that subshell, or even just its STDOUT, then you're stuck.

If you take the time and effort to understand how bash wants you to think, you can learn how to right elegant scripts that are as maintainable as anything else.

I've written a lot in bash over the years, and I feel like I understand it pretty well. But I would never say that even elegant bash scripts are as maintainable as "anything else". It is a clunky programming environment, born of compromises, with many traps that are easy to miss in code review.

There's `shellcheck`, as part of Syntastic, for vim. I find it quite useful. You could call it a Bash linter.

Shellcheck is great and is a standalone cli program, as well as a website if you don’t want to install it locally. https://www.shellcheck.net/

I'd rather always check my code locally if the alternative is pasting it into a random web from.

On the other hand, I wonder if the web page keeps track and could offer "best of bash mistakes".

Well, you can. From the website itself:

"ShellCheck is...

* GPLv3: free as in freedom

* available on GitHub (as is this website)

* already packaged for your distro or package manager

* supported as an integrated linter in major editors

* available in CodeClimate, Codacy and CodeFactor to auto-check your GitHub repo

* written in Haskell, if you're into that sort of thing. "

Sometimes you are not in your computer, the script does not have private information (i.e. open source or something you don't care to be public). Sometimes the website is simply more convenient.

*edit: formatting

A random web form can exfiltrate the data you paste into it (and whatever your browser lets it gather). A local program can exfiltrate ... approximately everything of value on the machine?

That's why Linux users typically install things from their distro's package manager. The bar to get malicious software in there is very very high (though it is not impossible).

But if you're still using Windows, then yes, I agree.

While I understand the sentiment, I'm not sure how bash could ever be as maintainable as a something written in e.g. Python (or even better, a strongly-typed language).

The thing with bash is, it's great for tying things together and quick bits and pieces, but it's not set up for writing maintainable code. Arrays, functions, even if statements comparisons can all be done in bash (as first-class features), but are just... easier in other languages. And then think about the refactoring, linting, testing tools available in bash vs other languages. And then on top of that, there's the issue of handling non-zero return codes from programs you call; do you `set -e`, and exit on any non-zero return code even if you wanted to continue, or not `set -e`, ignoring any errors as your script just continues.

Personally, when I feel I want to use a function (or array, or other similar, non-trivial thing), in bash, it's time to reach for another language.

Having said that, there are some nice programs written in bash. https://www.passwordstore.org/ being one that comes to mind.

> e.g. Python (or even better, a strongly-typed language).

Python is strongly typed. Maybe you meant "statically"? (As opposed to dynamically.)

Yes, you are right, and statically is what I meant, thanks for the correction.

> Personally, when I feel I want to use a ... non-trivial thing ... in bash, it's time to reach for another language.

Then you must not be writing any bash at all. Functions are useful for almost anything beyond a one-liner script.

Typical functions used in innumerable non-trivial scripts:

* print_usage()

* die() (see: https://stackoverflow.com/q/7868818/1593077)

:-)

You are kind of right, I don't write much bash, but I do write some simple scripts that I can call quickly and easily (e.g. start this program with these args, write the log file here with the filename as the current date, etc). Although regarding "Then you must not be writing any bash at all"; I'm not sure how you could have deduced this!

With regards to `print_usage()` and `die()`, yes, I would reach for Python 3 then. The `argparse` module and `throw` are first-class members of the stdlib/language and are better and more standard between programs than if I threw together these myself (and with `throw` you get a stack trace, which is nice).

Can you point me towards an elegant bash script? I honestly have never seen one that is more then a few lines.

I like to think this thing I wrote isn't so bad and is fairly readable, even if you don't know anything about shell programming:

https://github.com/GavinRay97/hasura-ci-cd-action/blob/a9731...

This is out of necessity. I'm not the sharpest tool in the shed, so I have to go out of my way to write things such that when I come back to them in months or years, I still understand what they do.

For this same reason, is also why I never use shorthand flags for scripts.

I have no clue what IE: "-s -y -o" might do (or even worse, "-syo", dear god my eyes! Also -- is that one special command, or a series of individual commands?!).

But "--silent --assume-yes --output-file" is pretty easy to grok immediately.

I agree with you that it seems readable, nice job on it. But there is comment there that says

Oh man this is so ugly

Not something I would expect in an "elegant" script. Don't think that it is your fault, just it is very hard to write anything elegant in bash.

Ah yes, that section.

Hahaha -- fair point. Readable: maybe. Elegant? No. Bash is far from elegant =P

I would add a variable ENDPOINT, initialised to "" and set to " --endpoint $THEENDPOINTVALUE" if the endpoint value was passed. Then include that in every invocation?

Sorry if I missed something in the logic, reading on my phone, but from the comment, this feels like something I do frequently...

Or, (thanks, Amazon): `--no-paginate` `--no-cli-pager` (dang that pisses me off! thank goodness for `jq`).

Sorry. "elegant" here triggered.

https://ilya-sher.org/2021/03/19/running-elegant-bash-on-sim...

I write a lot of bash, and can kinda agree that it’s full of footguns. The most of them are not even in bash, but in Unix tools, which you have to use due to the lack of standard libraries.

Somehow I noticed more footguns in bash than in utilities.

To have variable typos result in errors with `set -u`, then expand an array that is defined but may be empty, I need write it as `${ARRAY[@]+"${ARRAY[@]}"`. (Further explanation: https://stackoverflow.com/questions/7577052/bash-empty-array...) But maybe bash arrays are too new a feature, and should be avoided, so let's look at something simpler, like command-line arguments.

Parsing command-line arguments is easy, but parsing them correctly is ridiculously hard. `getopts` doesn't handle long flags, so it's out. `getopt` doesn't handle arguments with spaces in them, unless you have a version with non-standard extensions, so it's out. You're left with manual parsing, and it's a royal pain to make sure you handle every expected case (short/long flags, clusters of short flags, trailing arguments to be passed through to a subprocess, short/long options whose value is in the next argument, long options whose value is after an equals sign, and several others I'm probably forgetting about). And this is just to get the arguments into your script, before you actually do anything with it.

I agree that bash has effective idioms, and learning those idioms can make scripts easier to write. I strongly disagree that bash is "as maintainable as anything else", and scripts beyond a few hundred lines should be rewritten before they can continue growing.

Bash `getopts` handles short and long arguments gnu-style without any problem. The following code handles args like "-h", "-i $input-file", "-i$input-file", "--in=$input-file", and "--help":

  while getopts :i:h-: option
  do case $option in
         # accept -i $input-file, -i$input-file
         i ) input_file=$OPTARG;;
         h ) print_help;;
         - ) case $OPTARG in
                 help ) print_help;;
                 help=* ) echo "Option has unexpected argument: ${OPTARG%%=*}" >&2; exit 1;;
                 in=* ) input_file=${OPTARG##*=};;
                 in ) echo "Option missing argument: $OPTARG" >&2; exit 1;;
                 * ) echo "Bad option $OPTARG" >&2; exit 1;;
             esac;;
         '?' ) echo "Unknown option $OPTARG" >&2; exit 1;;
         : ) echo "Option missing argument: $OPTARG" >&2; exit 1;;
         * ) echo "Bad state in getopts" >&2; exit 1;;
     esac
  done
  shift $((OPTIND-1))

You can use bash-modules arguments library to generate argument parser for you.

Thank you, I haven't heard of that one before. However, all the examples look like it gets called as an external library, which would either need to be distributed with a script, or to already be installed by a user.

Is there a feature I'm not seeing that would generate an argument parser without external dependencies?

Yes, arguments::generate_parser function. Install bash-modules, then run `. import.sh log arguments`, then call `arguments::generate_parser '--foo)FOO;String'` , and it will generate function with parser, which include parser for foo option for both `--foo VALUE` and `--foo=VALUE`:

  ...
  
        --foo)
          FOO="${2:?ERROR: String value is required for \"-f|--foo\" option. See --help for details.}"
          shift 2
        ;;
        --foo=*)
          FOO="${1#*=}"
          shift 1
        ;;
  ...

One way to answer this would be to find the largest (in terms of LoC) Bash script you can find and try to add a feature or fix a bug.

But in reality you never see such scripts.

> Bash is not a "real" programming language, so do not treat it as one.

Is there a reason we aren’t using a shell with a proper programming language for scripting?

I can’t say I’m a fan of bashscript. I wonder why we are using this weird language.

Because cross-OS (not just cross-linux-distro but even that) standards-making doesn't exist anymore in a real non-broken way, and we're stuck with whatever standards of 20+ years ago. Whether official or de facto (technically bash is just de facto although `bash --posix` or `sh` is an official real standard). There basically isn't any "innovation" happening in this space in a way that could really result in anything as pervasive as bash. Maybe also cause it's just not "interesting" anymore, the 'sexy' things are many levels of abstraction higher than they were when bash came to dominance. It feels like now we're just stuck with it forever, indeed.

Maybe it's time for some other body --- say, freedesktop.org --- to start standardizing things that have traditionally been in the domain of POSIX.

After all, HTML got much better when WHATWG took W3C's toys away and actually started innovating HTML again. Maybe it's time to do something similar with Unix: I'm sick and tired of people telling everyone that we have to stay in some 1995 time capsule because we need to follow The Standard and if The Standard hasn't changed, too bad.

You say "don't exist anymore" but when was it better? Pre-Mac-OS-X you didn't even have an easy to access shell there on Macs, and Windows wasn't bash either. So if there was a golden era for that, it never included the two predominant desktop OSes.

De-facto standardization has at least brought us free officially-supported options for running Bash or other *nix shells on both of those platforms(if terribly outdated on Mac).

I agree with your broader point that we seem stuck with old shells and no contenders seems in a position to replace them. However some are certainly trying, like osh [1] that takes backwards compatibility with bash seriously, which makes adoption easier.

Maybe it'll be like the C++ vs Rust situation?

https://www.oilshell.org/

My feeling is that it is about convenience when typing at the terminal. Having a unified language for both scripting and entering commands is quite convenient. Also, bare strings while typing at the terminal is very convenient! It would be really annoying if we always had to add quotes for string args when we're typing at the terminal:

  $ cat "file.txt"

...and I don't even know what options would look like... maybe:

  $ ls "*.txt", ["l", "a"]

I dunno... this would mean we'd probably need parens now for precedent-type concerns:

  $ ls("*.txt", ["l", "a"]) > "file.txt"

Anyway, that is why I understood that it's this way. I can't point to a resource about it, though.

You look like you are just about to discover Powershell.

Powershell is how I instrument cross platform CI scripting, with the power of the module concept and manifess now you can write real powerful type safe apis that can run anywhere. It has its quirks though...

Any gotchas with running the same scripts on Windows and Linux? I've been thinking about switching to PowerShell. Love it on Windows. Never tried it on Linux but I did try it on macOS a few years ago and remember some quirks. Currently a bash (over)user.

One gotcha is powershell core has some different apis then the 'windows powershell', best to deploy core on both systems to unify the api.

Also powershell has no support for automatically handling external shell commands that error, you must manually check $LASTEXITCODE (easily done with a custom invoke api that you use for everything).

Also well, some of the escaping in strings is really odd...

You have to add quotes in bash if the filename contains a space. Variables should also be quoted for the same reason, because their values might contain spaces. "${var}"

Only in a script. On the command line, tab expansion will escape any special characters in a filename.

don't forget to handle filenames with newlines in them as well!

> It would be really annoying if we always had to add quotes for string args when we're typing at the terminal

With bash (and I suspect with every other popular shells out there), it actually means something different. In your second example, if you don't use quotes, the shell will do the extensions, but if you do, the process will have to do it (and in the case of ls, it cannot).

In some case it's good to have the shell taking care of it, but sometimes (eg, when using 'find') it's not.

My examples are a made-up idea of what a “real” programming language would look like in a shell, not actual bash.

Sorry, clearly my comment wasn't very helpful.

I was trying to point out that quoting parameters or not could bring a different meaning for a shell (as one could see it with Bash), with the implied consequence that one would have to carry over that nuance somehow to a shell where all parameters would have to be quoted.

I feel like this comment is almost a kind of inverse of https://news.ycombinator.com/item?id=23424853 (which I remember to have found interesting).

ruby in some cases feel just like shell scripting, even more than perl.. you can even do piping to chain two commands together https://stackoverflow.com/questions/4234119/ruby-pipes-how-d... (edit: using the shell library https://github.com/ruby/shell so, not out of box in ruby, unfortunately, which is a mortal sin for replacing bash in scripts =/)

Here are the ways to run a program in ruby https://stackoverflow.com/questions/7212573/when-to-use-each...

Stephen Bourne was a great fan of ALGOL, and had C preprocessor directives to "ALGOLify" C.

https://research.swtch.com/shmacro

The Bourne shell thus adopted ALGOL syntax, distinct from anything else in UNIX.

David Korn brought a pile of C, and wedged it into a max 64k program space (for Xenix), and his upward-compatible Korn shell (ksh88) had many, many features.

Then there were the "UNIX Wars"...

https://en.wikipedia.org/wiki/Unix_wars

The fallout of this was the choice of a few Korn features, but not all (no arrays, coprocesses, [some] eval, regex, et al.)

The wars defined the POSIX shell. Yes, it can be infuriating.

It's because Bash is everywhere and is thus the next least common denominator after sh.

Bash is not everywhere.

There are two very distinct places where Bash is not, that I have found.

Bash is not in busybox. It pretends to be, but what is really there is the Almquist shell, with a bit of syntactic sugar to silence the common complaints.

Bash is also not in Debian's shell (/bin/sh). In that place, there is no tolerance of bashisms.

It is important to know the POSIX shell for these reasons.

Bash is readily available everywhere, though. Is there a widely used OS that cannot run bash, but can run BusyBox ash or dash?

Bash is not used in Android.

The Android userland explicitly avoids any GPL software.

The Android system shell is the MirBSD Kornshell (mksh).

This is (obviously) a massive platform.

But, you can nevertheless install bash on Android. That was my ask -- show me a widely-used OS that cannot run bash but can run other similar shells.

> Bash is also not in Debian's shell (/bin/sh). In that place, there is no tolerance of bashisms.

When writing a bash script, shouldn't you be setting the shebang to /bin/bash which is available on Debian?

Bash is the default interactive shell, but the Almquist shell is the default for /bin/sh, and must be used for all system startup activity.

It is otherwise advisable when speed or portability are required.

If you want to choose a programming language, there are hundreds to choose from. You could write a small program ("script") in Brainfuck. But that would be really annoying, and it would take you a long time to get anything done. Other languages may also take a long time to write, test, execute, and maintain. And they may be better at some things than others.

You have to step back and remember the point of all this. Why are you programming? To solve a problem, and make a human's life easier. What is the problem you are trying to solve? How can we make the human's life easier? In this case, it's "I want to combine bunch of programs together in a command-line interface, to make it easier to use these programs to get my work done on the command-line." So, what solution should we choose for this scenario?

Lots of different "scripting" languages exist (we used to call them "glue languages"). Today Python is the most popular, and before it was PHP and Perl. But who cares if they're popular? What are they good for? Why would you choose one over the other?

Python is a general purpose language which is easy to learn, easy to write, and easy to read. Well that sounds nice, but it's very "general" and not specific to our use case. PHP is a language designed for use in web development. Perl is a language designed for system administration-type tasks.

Bash scripting is designed for making it easy to combine already-existing programs in a command-line shell, with no modification or creation of programs required. The only "dependency" is the Bash interpreter (which happens to also be the command-line interface we started our problem with! how convenient!) and an existing collection of software tools that work with each other through a command-line interface. Every part of it is explicitly not designed to "make programming easier". Instead, it is designed to make "combining programs in the command-line" easier. The result is things like every single word you type is potentially just an outside program being executed, or arguments to that program. No other language has this quirky behavior, because they're not designed solely to make command-lines easier to use.

I know Perl well, and the language is fantastically useful as a system scripting language. It combines some of the best features of common shell scripting programs with more programming language-specific features. It is designed with shortcuts and "magic" to make quick work of command-line scripting tasks. But ultimately Perl was not built into the command-line, so its utility is always a bit stilted. Going between the shell and the Perl code and back to the shell isn't as flexible as if the Perl code was embedded in the command-line. Perl also doesn't have as simple a facility for pipes as a command-line shell does. Oh, and a Perl install is rather large, isn't available everywhere by default (anymore), and has all the usual dependency problems. So even though I know Perl very well, if I need to just combine existing programs in the command-line, I always use Shell scripting instead of Perl. (There are Perl shells which solve this problem, but then everyone needs to learn Perl, and Perl can be.... idiosyncratic)

You could also use Awk for a large number of general programming tasks, but again it's designed for a different specific problem: pattern-scanning and processing, not generally combining programs together.

So we use shell scripting to solve the problem because it was designed specifically for our problem, it's ubiquitous, and simple to use. You could use any other programming language, but it wouldn't fit our scenario as well. Once your use case changes, and you are no longer only trying to combine existing programs together, or the existing programs don't do what you want them to do, then you need to use a different language that better fits that new use case.

> Is there a reason we aren’t using a shell with a proper programming language for scripting?

Mostly because the people who want to introduce a "programming language" into the shell don't prioritize being a shell.

Check out the "Oh" shell for contrast. This is what a programming language looks like when you force it to conform to being a shell first priority.

https://github.com/michaelmacinnis/oh

https://www.youtube.com/watch?v=v1m-WEZz46U

This is "Scheme-like" (actually the late John Shutt's Kernel-like) but has FEXPRs so things can be redefined and evaluation can be controlled.

Perl used to be used a lot to automate things. I feel Python has taken some of that, but also have seen Bash being picked up more.

I tend to start writing things in bash (piping into things like grep, tac, sed, etc

However if I’m not careful they get painfully complex and I tend to regret not writing it in perk. As such I now switch to using Perl when I get to the function stage, or anything other than the most simple bash arithmetic.

I don't know why "we" are. Just use Python. Or whatever language you're using for your main system (I've written "scripts" in Scala, sure it takes a second or two to warm up the JVM but it's honestly fine). Or TCL if you really must have a language that can be used as a login shell.

shell is good enough, when people started to use script language, they just wanted to fill the empty between bash and C, no one want to replace it, so bash and other shell still there

There's csh.

In about 1995 I asked the question "how can a shell intended for C programmers not support functions?"

I've still not found a suitable answer.

  > Getting anything out of a subshell that isn't from STDOUT is impossible.

Yes that is a known limitation that has horrible, horrible workarounds. Don't use them.

That said, for scripts whose main input _is_ from a file or STDIN, and also whose main output _is_ to a file or STDOUT, then bash is far more often the right tool for the job than it is given credit for. Of course, I'm talking about bash and a host of other utilities that are often packaged with it such as awk, grep, sed, cut, etc.

For processing text I find myself often choosing between bash and Python, and very often if I choose bash I'll feel that I've made the right choice.

"So nobody uses subshells."

djb has always used them, even in the shortest of scripts.

Otherwise I agree with everything in this comment.

There is a reasonable argument to avoid using bash for non-interactive scripts. The benefits of any additional bash features, so-called "bashisms", arguably do not outweigh the costs of making these scripts non-portable. One of the many advantages of shell scripts is that they are portable and have tremendous longevity; shell scripts can last a long, long time. There are no version changes and aggressive feature creep to worry about as is routinely the case with programming languages. The scripts just keep working, every day, and we can forget we are even using them, e.g., they are being used in the various ways by UNIX-like OS distributions.

One of the today's programmer memes was "Get Stuff Done". Maybe the shell was not meant for today's programmers. But for "sysadmins", or "DevOps", or whatever term anyone comes up with in the future, people who can "administer/operate" computers running a UNIX-like OS for themselves or for someone else like a client or an employer, the Bourne shell works for its intended purpose, better than anything anyone has come up with in the last 50+ years. There's a lot of stuff that "gets done" with the shell, whether it is on someone's own computer, their client's or their employer's.

Attempts at "shell replacements", cf. alternative shells, usually look like interpreters for programming languages, not shells. Perhaps this is not a coincidence.

Bourne shell is relatively small and fast. Computers with small form factors often use UNIX-like OS and when they do they usually include shells. Many programmers seem to dislike the notion that such a layer exists. They often try to blame the shell instead of their own lack of interest in learning to use it.

The shell is boring, and "boring" is sometimes the wisest choice. Most software has expanded to consume available resources (often the developer's choice not the user's). This makes computer performance gains difficult for the end user to discern, e.g., decade after decade, routine tasks seem to take the same amount of time. However the shell and many "standard" UNIX utilities have not changed as much in the same period. Subshells may have been "slow" many years ago. IMHO, they do not feel slow today. Routine tasks performed with the shell seem to run faster, as one would expect after hardware upgrade.

I posit: Life is too short to learn every programming language du jour but it is long enough to learn the Bourne shell, reasonably well.

"Shell" as used here means the Bourne POSIX-like shell, such as NetBSD's Almquist shell, not userland utilities that the shell may or may not call in a script. Scripts can of course test for differences in utilities where there is uncertainty.

If scripts written for a POSIX-like "lowest common denominator" shell were not portable, methinks projects written for such a shell, such as autoconf, would not work on so many diffferent systems. The amount of free software found in open source OS repositories that is built using "configure" shell scripts is not small.

Debian and other Linux distributions use a shell derived from NetBSD's Almquist shell. People who design and maintain these operating systems suggest that, for non-interactive use, this Bourne shell is faster than Bash.

One of the principal things that "got done" using Bash (at least until 2017 -- after that, I couldn't say) off the top of my head was the GFS at NCEP. Yes, it was predominantly Fortran for number crunching; but what tied all the individual programs together was a super massive, maintained, and constantly modified and improved Bash script. And yes, there were plans in the works to switch out for something else (which is why I cannot say, four years later, what the status quo is). I should also mention, it was this Bash script that ran parallelized on the supercomputer -- and it was ultimately this script which, every three hours, produced the next set of forecasts, 24/7.

Bash scripts and shell scripts are not portable.

They have to shell out to other commands in order to do something meaningful.

Depending on your OS, these commands may not exist, or have a different set of commandline flags.

Well you can use arbitrary file descriptors to get things out of a subshell, not just stdout. I had to do it when I had to get something out of band, while redirecting stdout through a pipeline.

Mind, if possible, it is even uglier than using stdout.

yeah, I was wondering if there was some reason that something like this wouldn't work:

  err_f="$(mktemp)"
  third_f="$(mktemp)"
  out="$(cmd 2>"$err_f" 3>"$third_f")"
  err="$(cat "$err_f")"
  third="$(cat "$third_f")"

Seems like that should work fine, and it's not even that odd looking or hard to understand. But I haven't tried, not sure if there's some surprise lurking in the depths of tempfiles or something.

Normally I use mkfifo then immediately rm it after opening the file with exec.

The fact that there isn't a good way to open an anonymous pipe and get its two file descriptors in bash is very annoying.

Few people know about `{varname}` I/O redirection...

Yes. The fact that shell scripting is both so powerful and yet such an ugly language is painful.

You're right about the disadvantages of subshells, but wrong about this:

> avoid bashisms

What's holier-than-thou is insisting that people stick to a dialect of shell scripting that hasn't changed in decades because some people think that conforming to POSIX is its own reward even if it makes life harder and programs worse.

No, thanks. I'll stick to bashisms. Things like coprocesses and filemap make certain classes of program much easier to write. And yes, shell scripts are programs, and the quickest way to turn these programs into unmaintainable balls of mud is to follow your advice to avoid normal programming best practices simply because the program you're writing happens to be a shell script.

Here is a script that shows exporting results from a subprocess to a parent shell using a temporary file:

  #!/usr/bin/env bash

  process_line() {
      if [[ $* == *foo* ]]; then
          echo "found_foo=1" >>$IPC
      fi
  }

  IPC=$(mktemp)
  find . -maxdepth 1 -type f | while read line; do
      # note: this is run a subprocess
      process_line "$line"
  done

  source "$IPC"
  echo "found_foo=$found_foo"

> Bash is not a "real" programming language, so do not treat it as one.

Bash is definitely a different paradigm to your average imperative or functional language and thus requires a different approach but I wouldn’t go so far as to say it’s not “real”.

It certainly fits the criteria of a programming language even if it does have more warts than a pantomime witch.

Yes.

Compared to other languages, Bash can be incredibly worse for some things, but much better for others.

People who don't realise that won't be able to get the best out of their toolset.

It is Turing Complete...

I don’t think it makes much sense to judge software by whether it is Turing complete because otherwise that would make Minecraft a programming language

So are you deprecating the term "Turing Complete"? Mighty bold move there. And on that same note, have you seen the stuff being done in Minecraft? It is quite up for debate whether it is a programming language.

> So are you deprecating the term "Turing Complete"?

No. It’s still an interesting yard stick. It just doesn’t describe programming languages in full. For example, and someone counterintuitively, it is actually possible to design a programming language that isn’t Turing complete. Some esoteric functional languages do actually fall into that category. But generally programming languages would be a subset of Turing complete software.

> And on that same note, have you seen the stuff being done in Minecraft?

I have, hence why I cited it as an example.

> It is quite up for debate whether it is a programming language.

One could debate anything but it doesn’t mean the argument is made in good faith.

Minecraft is a game. I’m happy to even extend the definition and say it’s supports some visual programming mechanics. But that doesn’t mean it is a programming language.

Just because something has 4 legs and a tail, it doesn’t automatically mean it is a dog.

Yep bash is not a real language.

If anything you are saying it should be deprecated in place for a real language. If people want to do more advanced things why does the basic language of their shell not support it? Why should you not care about the correctness of your shell scripts which can run important things? Seems a perfect place for a functional language. It’s high level, you can just declare what you want. I imagine your interpreter can add in all sorts of checks to help make your scripting more correct.

Powershell attempts to use a more scripting friendly language in the shell.

And it’s a horrible shell experience.

It’s great for scripting (but not as good as other scripting languages) but it’s a terrible experience on a day to day basis, with the only salvation being the existence of 2-3 letter aliases which make things slightly more manageable.

Bash is a great shell language that whose scripts are useful because it allows you to trivially convert your manual shell actions into an executable script.

If you need to do anything complex (or rather, something you probably wouldn’t type into the shell manually if you had to do the thing on a one off basis), then you’re probably better off using a different scripting language.

But changing BASH with scripting as a priority would probably make it a worse shell language (which is not to say there are no improvements to be made…there are).

No need for array when i can create files and combine their content with commands like tac and other coreutils text utilities.

> So you can't define an array in a subshell and then use it outside the subshell, and you can't return an array (or anything that isn't a string) from a subshell

Huh. How does one return a value from a bash function? I've always only ever seen the "echo calling convention" used in scripts, or use of global variables.

Bash does have `typeset -n` (from ksh) which allows shell variables to be passed to a function by reference.

Yes, and which gets you Tcl-ish pass by name semantics.

Cool, thanks!

Just don't use Bash.

A lot of the time you don't even have the luxury of assuming "bash" is present, you've got only "sh".

Since all valid "sh" syntax is valid "bash", you've got to restrict yourself to only sh-valid code in the event the host doesn't have bash.

I have gotten used to doing this -- these scripts usually wind up being run in Docker containers that don't have "bash" installed for the sake of minimalism.

IE, you can't use [[ $predicate ]], but instead [ $predicate ], etc. Lot of subtle differences.

Not all sh syntax is valid bash — in bash [[ is reserved but not in sh. And if you want syntax that is valid but doesn’t do the same thing in sh vs bash there’s plenty of examples.

You can't ever assume that /bin/sh is a POSIX shell so then you get limited to just an early Bourne subset.

I would agree with this and use python instead, but for uuhh "python problems with strings". IF you are 'blue sky' open, making new things, yes, sure. no BASH. But for things I built 8 years ago that are non-trivial, and I don't prioritize an entire rewrite, BASH works very well. And guess what, every two years that 8-years-ago just comes along with it. BASH is here to stay.

The POSIX shell can sometimes achieve with a few lines what would take dissertations to do in C.

The supersets of POSIX are often even more effective.

Ugly as the tool may be, in the box it stays.

If you want to program, use a programming language.

Sir! Yes, sir!

throwaway984393 <-- just a troll...

Edit: It seems I should elaborate.

> Getting anything out of a subshell that isn't from STDOUT is impossible. So you can't define an array in a subshell and then use it outside the subshell, and you can't return an array (or anything that isn't a string) from a subshell. If you only use subshells and want to use any kind of data structure that isn't a string passed from STDOUT, you have to do it globally.

Yes, but that is no problem at all, because that is the way shell scripts work and if you do it in a functional way, where the function has no state, it is great (we a few exceptions).

> And subshells are slow. So nobody uses subshells.

Shell scripts are slow in general, subshells don't make an exception but also don't have a large impact. And people do use subshells.

> If you use Bash for programming, you have to stop thinking in terms of the holier-than-thou software engineer, whose ego believes that a superior, "clean" design makes a superior program. You should embrace globals. You should switch between using or not using the enforcement of set variables or program exit status. You should stop using Bashisms and subtle, obscure language features unless you absolutely have to.

If you use Shell scripts, you should understand that this language has been designed decades ago and that professionals advise to use it just in short scripts to connect binaries.

> Bash is not a "real" programming language, so do not treat it as one. Do not look for hidden features, or try to do things in cute ways that nobody else uses. There is no superior method or hidden knowledge. Just write extremely simple code and understand the quirks of the shell.

That part is mostly okay, but "real" doesn't get the point, as it is a real language, just one with many problems. However, given its superior strength we haven't overcome it yet...

> However, given its superior strength we haven't overcome it yet...

"Superior strength" is probably just a matter taste, however I would really like to read your explanation on why you think so.

Well, if it would be just bad, why do we still use it? Why don't we use e.g. Python (which is a much better designed programming language)?

Some reasons why POSIX shells are great:

- they are part of POSIX (a Standard which is implemented by multiple OS) and therefore often available

- you can use them to compose programs written in different languages

- you can easily connect the output of one program with the input of another (pipes)

so it's not Bash by itself which is great, it's POSIX shells?

> they are part of POSIX (a Standard which is implemented by multiple OS) and therefore often available

That's not an intrinsic property of Bash.

> - you can use them to compose programs written in different languages

> - you can easily connect the output of one program with the input of another (pipes)

This last point subsumes the previous one. So basically, you're saying Bash is great because it's a shell?

Basically, yes. Most arguments expressed here apply to Bash as well as to other POSIX compatible shells. Even the trick from the initial blog post is also works for POSIX shell AFAIK (didn't check). Definitely working is something like (which is just a little more verbose):

  myFunc() { (
    #...
  ) }

So in this case I don't think it makes sense to make a distinction between the two. In essence, I think Bash and the other POSIX shells share most of their greatest strengths and weaknesses.

But, wasn't the original argument discussing the merits of Bash as a shell? So you're saying that Bash is great as a shell because it's a shell?

My understanding was, that the initial argument was about Bash as a programming language and most aspects discussed were aspects were Bash is no different than POSIX:

1. Return values of subshells 2. Clean design 3. being a "real" programming language

The only Bash specific part was

> You should stop using Bashisms and subtle, obscure language features unless you absolutely have to.

But in fact, the usage of subshells to limit the scope and subshell return values have nothing to do with Bashisms.

Reply on news.ycombinator.com