The temptation of writing shell scripts, illustrated

Bit of questionable advice: I've found using jq for all the argument mangling/postprocessing/glue tasks instead of sed, awk etc to be pretty workable lately.

The -R switch reads in arbitrary text instead of json and presents it as a sequence of json strings internally.

The capture() filter applies regexes to the string and extracts subgroups (which will be represented as json objects).

The gsub() filter performs search-and-replace on a string.

Finally, the -r switch writes the result back to stdout as a raw value, removing any json quoting and escaping.

By default, processing is linewise, but there are options to process the whole input at once instead. There is also supposed to be a way how you can have per-line processing with state tracking across lines (like sed's "holding space"), but I haven't tried that out yet.

I found this approach often more flexible than sed and easier to reason about. Also, I believe it may be faster, as more things are happening in a single process.

The one exception: grep on files is still orders of magnitude faster than anything else. I believe there also was an article about this on HN a while ago.

So if you have to find some data in a larger collection of files and then have to turn the data into something structured, a good approach is to make a "coarse" query with grep or grep -R first, to narrow down your set of candidates, then postprocess your remaining candidates with jq.

Also, apropos single process: Another questionable advice is that for/while loops in bash are subshells and you can use them as part of a pipe chain. This often lets you reduce the number of processes you need.

E.g. instead of

  for f in $files; do foo "$f" | sed ...; done

You can often write:

  for f in $files; do foo "$f"; done | sed ...;

Which will only launch a single sed process for all files instead of one per file.

But yeah, shell scripts do have many shortcomings and in particular the error handling situation seems extremely hard to fix. So actually sensible advice is still: Use python.

I think people are underestimating the power of `bash+curl+jq`. In pretty much every scenario where I want to consume an API, I just fire up my vim+shellcheck+shfmt setup, split my logic into functions (using `local` for private variable, mind you), then just spam `get_endpoint | jq filter`s in a `getopts` loop, and bam, suddenly you have a clean and concise CLI.

Granted, Bash is quirky. So I should say I program in bash+shellcheck not just Bash.

And Jq. Jq is a functional programming language in itself! I do regex or string mangling inside jq filters.

Why should I use Python over this? The moment you want a clean and cool feature that Python has over Bash+jq, you should incorporate third party libraries. And adding just one non-std lib, means you have to resort to `virtual env`. And why not adding also `click` or `requests` or .. and now you have ~10 moving dependencies. Let's not talk about Python's subtle differences between versions.

Compare it to Bash-as-glue-of-unix-tools approach. You want image processing? Just use `imagemagick`. You want web scraping? `pup` is your friend.

ProTip: do `my_command || exit 1` if my_command is bound to fail. Or `if ! my_command; then printf >&2 "[ERROR] reasons"; exit 4 fi` for more elaborate exception-handling.

Perl seems to be fading away, but for a long time it was the obvious tool to use when your program got too big for a shell script.

> Python, Go and so on need more structure and often don't have quite as simple and convenient methods of doing various shell like things.

This is Perl's niche, or at least it was Perl's first niche before it started getting used for CGI. The things you'd do in a shell script are generally convenient in Perl too, the main inconvenient part is selecting from the many different ways to invoke a program in Perl, like system(), backticks, qx//, open(), etc.

Perl is also more commonly installed and has faster startup time than Python, in my experience.

The “best practice” of preferring higher level languages is questionable when it comes to System level tasks such as this one.

If it’s something that needs to be part of a complex system, sure higher level langs can be powerful. If it’s a standalone thing, shell scripts are generally ok.

In fact higher level languages add complexity: their build process, library updates, breaking changes of dependencies all make it a pain to maintain. Shell scripts? “ls” is never gonna change.

IMO the biggest reason shell scripts are avoided is simply because most software developers aren’t very comfortable with the syntax (tbh it does feel archaic in certain places) or know how unixy stuff like signals, pipes etc really work. If you have to do system level tasks though it’s probably worth the effort to understand these things.

I think one of the things that still make shell scripts so alluring is that it's the same language you use for your shell.

Many of my scripts for personal use were basically just cleaned up and slightly extended versions of the command sequences I typed in anyway. This is different from writing a script in a proper language where you usually start from scratch.

I believe, this was Perl's approach originally: Let you use your shell commands as a starting point and then gradually extend the code in a safer language. However, this still leaves you with many of bash's syntax issues.

On the opposite side, many modern languages do have REPLs - but I have not seen anyone so far use them as actual day-to-day shells.

So I think to really get rid of shell scripting, you'd have to develop some syntax that would work well both for scripts and for shell use.

Pro shell scripts:

- I often encounter machines without Python or Go. I never encounter machines without Bash.

- Python and Go will probably change faster than Bash. So the maintenance cost of scripts in Python or Go will be higher.

- When you figured out something tricky on the command line, you can paste it into a Bash script verbatim.

- Bash is the perfect glue language. If the task at hand is to glue together some commands, nothing beats Bash.

Not sure if someone mentioned it already, but for such use cases Xonsh[1] is amazing.

It allows you to use shell stuff in a python script. Basically it allows you to use the best of both worlds, i. e. you can use python for the logic flow while retaining all your quick shell oneliners when neede. And it makes this interaction of python and shell super convenient, i. e. run your oneliner but process its output with python. Much faster than coding everything in python or in shell.

[1]: https://xon.sh/

Shell scripting is not bad these days. There’s a linter - Shellcheck - that will catch most gotchas in scripts, there are a lot more resources to learn shell online, and there are tools like Bats to make your life easier if your shell scripts grow too big.

Writing logic - functions, for loops, or if statements- is still very annoying in Bash. Likewise writing a piece of code that glues calls to external commands is extremely clunky in almost all other languages (except Perl an Ruby), notably Python which is often treated as a Bash replacement.

Finding the right balance can be tricky, but choosing only Shell or only Python is not the right way to go.

Shell scripts require more direct systems knowledge than languages with an abstracted approach to _any_ system. As a rule of thumb if I am using python to drive (popen) system binaries I don't write in python and if I have significant data structures I avoid the shell. TBH the popular movement and younger crowd have kind of dismissed aptitude with the traditional unix shell/awk/sed as not worthy of significant effort due to any number of (mostly shallow) complaints. (gnu) Awk is a wonderful language in itself but sadly most users don't know anything about it and use it for trivial operations. These days if I am working with younger people in a team environment I default to python and avoid the bickering. That's the way wars are won - make it unpleasant enough that surrender is a better option.

Shell are old classic human-computer UIs, so they are battle tested, well known, daily used by almost anyone who can write a script. That means: scripting is natural for a vast cohort of users, it's natural because while weird in language terms it's a glue of a set of known and frequently used tools.

There are much more shell users than Go, Python, ... programmers. Also those who happen to be programmers normally use certain languages, typically more than one, typically far more diverse than just most used shells, for a certain kind of activity, like programming in a team a not-that-small project. As a result their mindset is not "quickly do something/quickly automate something for me, myself and I only, on my own desktop" but to be part of a larger game with need to cooperate, produce code easy to read by others, create a set of entry points for integration etc. Those programmers are not all, but mostly also shell users and they tend to use the shell not just for specific large projects but also for quick, simple&dirty personal stuff. So as non-programmers who use the shell they know and use it daily.

In the past we have had perhaps better tools, like Smalltalk on Xerox desktops or Emacs/zmacs lisp on LispM BUT their actual public is next to 0 and modern Emacs-ers are on modern desktops not LispM and they are typically born on modern desktop so they know the shell and in Emacs they'll wrap anyway countless shell-centric tools so end the end a script is often quicker and simpler...

Here's an attempt at the script. Let's golf it!

  N=10
  find /sys/fs/cgroup/ -name memory.current | while read f; do
    test -e $f || continue
    echo `cat $f` $f
  done | sort -n | tail -n $N | awk '
  match($2, /system.slice.([a-z-]*).service/, arr) {
    $2 = "/system/" arr[1];
  }
  match($2, /user.slice.user-([0-9]*).slice/, arr) {
    cmd = "id -un " arr[1];
    cmd | getline uid;
    close(cmd);
    $2 = "/u/" uid;
  }
  match($2, /cgroup.([a-z]*).slice/, arr) {
    $2 = "/system/" arr[1];
  }
  { print }'

Some people on this website keep insisting that you should use the 20 or so Unix DSLs to solve small problems, and then graduate to a general-purpose language like Python when you have a big one. I have pointed out one problem with this, which is that it means you have to learn a ton of arcania for each Unix DSL, and you end up with a Rube Goldberg machine of DSLs invoking DSLs. This post points out another set of problems. I now just use Xonsh (a Python dialect with some syntax sugar for shell things). Some people say that this might stop working after some updates, but I don't do this for a living, and those of you who likewise don't should consider following me.

If the project is just for "glue", then any reasons it shouldn't be used? I'm currently creating a "glue" project with shell scripts, and the biggest issue is that I'm creating a lot of environment variables.

I wrote a few scripts in Deno this week. In a single file I can import packages like markdown for parsing. It's typescript. I can handle arg parsing. Shebang works. This will be my go to for small local scripts

Show the shell script!

The only time I use shell is if I expect a pure list of commands with no loops or significant logic.

Actually, I'm not sure I've ever done a loop in bash at all.

The few times I've tried to write even trivial programs in bash resulted in me eventually encountering something ugly and unpleasant, then quitting and starting over in Python, and having a working solution extremely fast.

Has anyone else used https://github.com/google/zx here?

I've started using that instead of bash script and js/node/deno scripts. it's kind of a nice combination of things.

the only pain with this is writing to files. > doesn't just work.

shell: I don't give a care about anything.

posix shells with `set -euf`: I give one care.

perl, erlang: I care about arity.

python: I give one care about data type consistency.

java: I care more about types than command line interoperability.

ruby: I care about some things but not speed or memory or scalar/vector predictability.

lua: I care about everything but indexing.

c/c++: I care too much.

go: I care about concurrency.

rust: I care about making fun of Stroustrup.

haskell: I care about... whatever. I'll find out some day when I feel more eager.

javascript: Burn the world! Burn it to the ground!

bf: I care about obfuscation.

altJVM/altJS: I care about obfuscation--with data types.

.NET: I care about taking a paycheck.

cobol: I care about indentation even more than Python does.

random DSL's: I care about doing it all over again.

d: I care so much but nobody ever listens to me.

PowerShell: I care about COMSPEC environments.

non-POSIX shell: I care about the good old days.

assembler: No you don't.

HDL's: Yes you do.

php: I care about pre-Rails projects.

lisp: I cdar care less.

regex: You'll care when I break prod in a week.

“Master Foo and the Ten Thousand Lines” is all that needs be said

http://catb.org/~esr/writings/unix-koans/ten-thousand.html

> you shouldn't write anything much as a shell script and should instead use a proper programming language

why?

I had this problem, so I wrote a script. Then, I had 2 problems...

- old saying

Seriously, though, treating shell script as a prototyping tool can be really powerful. Don't be afraid to rewrite it again as a 'proper' program if you have concerns about maintainability, etc.