Hacker News Reader

Comment by jmmv | Hacker News Reader

I cringe when I hear people (often senior) saying things like: "Yikes, a 500-line shell script. We must rewrite it in Python for maintainability!!!!1!" -- without even looking at how it has been written.

Yes. Shell scripts can be terrible, and most are. But if one approaches them thinking that “the shell is a real language and must be written as such”, with solid principles in mind, it is perfectly possible to write clean, maintainable, and testable shell scripts.

Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway.

(Edited to add a couple of links to related stuff I wrote in the distant past: 1. https://jmmv.dev/series.html#Shell%20readability and 2. https://github.com/jmmv/shtk)

View on HN

From the article:

  consider this sentence:
    "arrays do not exist in posix shell"

  ...

  "NOOO!" you shout, because you like data structures a LOT
  "now THAT'S a good reason to use bash! bash supports arrays!"

  absolutely not!

  i believe that you should use shell if your problem is:
    - small and scoped (~200 lines of shell or less)
    - unlikely to increase in size and scope as time goes on
    - not very complex

  if you find yourself desiring arrays, good error handling,
  static typing, structs, etc, then your problem should
  be solved using a different language, not shell.

I tend to agree with this heuristic.

It's also why you see people bothering to use perl 5 at all in this day and age. It just happens to be preinstalled on a decent majority of *nix systems.

  A young traveller came to visit Emperor Sh, and found him sitting in his
  sparsely furnished temple.

  “Emperor Sh,” he said, “I am told you are the greatest scholar of shell that
  the world has known.”

  Emperor Sh made no reply.  The traveller continued.

  “I have come to ask your advice.  I am thinking of developing a character-based
  graphing tool.  It will interactively change the plot based on key presses.
  Which shell commands should I use?”

  “Don’t do it in shell,” said Emperor Sh, curtly.

  The young traveller was confused.  He tried again.  “Well… I am also working on
  a database audit script.  It needs to verify that certain characters do not
  appear in any fields in several tables.  What should—”

  “Don’t do it in shell,” interrupted Emperor Sh.

  The traveller began to despair.  “I have journeyed one thousand miles, tried
  thirteen distributions of my operating system, and waded through hundreds of
  badly-written manual pages,” he cried, “and now I have finally come to Emperor
  Sh, the greatest shell programmer in the world, and I am told to use no shell
  at all!  Perhaps I should do what my Python user friends told me to do, and
  just pipe together some scripts in venv environments!”

  “Good idea,” said Emperor Sh.  “Do it in shell.”

  Enlightenment crushed down on the young traveller and he bowed to the Emperor,
  sobbing with reverent joy.

https://sanctum.geek.nz/etc/emperor-sh-and-the-traveller.txt

Thanks for sharing that one!

The "unlikely to increase in size or scope" is the big one. It basically eliminates almost all uses of shell scripts (as it should).

I would add a 4th condition: you aren't going to share the script with anyone else (including CI) - it's going to stay strictly on your computer and nobody else will ever read it.

Perl and Python are part of LSB

https://refspecs.linuxfoundation.org/lsb.shtml

LSB support has been mostly dropped by some distros (e.g. Debian dropped support in 2015)

Yeah, but perl 5 hasn’t broken compatibility (as far as I’ve noticed) since 2000 or so. Python breaks compatibility every week, in practice. Does LSB include some sort of magical frozen in time version of python 2 or something?

In my 12 years of python I was hit with compatibility problems once, in a school project where someone started in python 2 and we changed for python 3. What libraries break that often so I can stay away?

scipy / numpy / etc broke multiple times for me.

I’m usually dealing with other people’s python, so I don’t remember which ones broke. Usually stuff around 3rd party services, async I/O or parsing.

So is the RPM format. I like LSB particularly as a reference for obscure ABI details, but I don’t think anybody actually builds either systems or applications to it.

I'm more concerned about whether or not people on BSD or OSX can use my scripts to be honest

> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway.

The dehydrated ACME client is 2400 lines of bash/zsh:

* https://github.com/dehydrated-io/dehydrated

And its external dependencies are OpenSSL and cURL. The acme.sh shell ACME client is 8000 lines of shell:

* https://github.com/acmesh-official/acme.sh

The official Let's Encrypt client is written in Python, and the core 'executable' is much longer, and in addition it pulls in a boatload of dependencies:

* https://packages.debian.org/bullseye/python3-certbot

I'm not sure why -- the one shipped with 9front is less than a thousand lines of C, and supports both DNS and HTTP challenges:

https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cm...

http://man.9front.org/8/acmed

The ACME protocol isn't that complicated.

Even with Python things can be written small (200 LoC) it seems:

* https://github.com/diafygi/acme-tiny/blob/master/acme_tiny.p...

* https://github.com/diafygi/acme-tiny

* https://packages.debian.org/stable/acme-tiny

100% disagree. Standard /bin/sh has terrible error handling facility, which on its own is enough to disqualify it as a "proper" programming language in my book. Even C is not that bad, and C is pretty terrible.

I'm proficient at POSIX shell scripting and use it all the time, but every time I discover that a tool I need to maintain or, god forbid, improve upon is written in pure shell I immediately get a migraine.

The shell is not a real language and must be written as such.

There's no such thing as "standard /bin/sh". You're talking about busybox ash, or dash, or something else that is shipped as /bin/sh in your distro.

Take a look at winetricks, or clitest, or crossroot-ng, or shellspec. They are modular, easy to read, well written pieces of software that are written in shell.

In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language. They were wrong, because they only saw browser differences and the issues with it, not the great things about it. The bourne shell is in a similar state.

I meant POSIX shell or more specifically a loosely defined subset of all these shell dialects that is reasonably portable in practice.

I'm not saying that it's impossible to do, I'm saying that it's very difficult, requires every contributor to be extremely well versed into shell coding and the particular convention of the codebase, and small issues can easily snowball out of control. It's doable, but it's almost never worth it IMO. One notable exception is when you need max portability with effectively zero dependencies.

>In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language.

They were partly right, which is why modern web development bears very little resemblance with early 2000's JS hacking. Remember that even JQuery only appeared in 2006. And those were the dark days of IE6...

But IMO that's not exactly the same thing: the issue with Javascript was less with its syntax and more with the very poor Javascript API and standard lib (or lack thereof), shell scripting is the other way around in a sense: you have access to all the power of the OS and all the programs in it, but it's the actual programming language that's lacking.

Actually a decent language that would transpile to POSIX sh would be interesting, now that I think about it.

The POSIX standard for sh is broken, it was born broken.

For example, all moderns shells support some form of `local` for declaring local variables. There are some small subtleties but consistent behavior can be achieved across interpreters. It's not on POSIX though. Shellspec complaints about it (but it shouldn't).

It sounds small, but it's a huge thing to have local variables instead of a huge global scope.

There are many other features that are widely available but with slightly different interfaces. Kind of what happened with ActiveX/XmlHttpRequest in JavaScript.

What is missing is precisely documentation on the subset of stuff that _can_ be portable with coding standards, polyfills and adapters.

I'm not talking about core utilities here. No `cat` or `grep` or nothing like it. Basially, `PATH=''`, only builtins are allowed. Even with this constraint, the shell is usable and powerful.

Unfortunatelly, most people are well versed in bash+coreutils, not raw portable shell. It requires learning a few things that are not straightforward at first (variable substitution, how the shell forks processes, making evals without making a mess, nested quoting, etc).

> Actually a decent language that would transpile to POSIX sh would be interesting, now that I think about it.

That would be nice. I believe it can be written in shell itself, self-hosted and portable. It's hard work though (on the same level of writing babel from scratch).

> Actually a decent language that would transpile to POSIX sh would be interesting, now that I think about it.

Isn't this how we got the ultraportable monster configure scripts? Not sure about "decent" though...

This is terrible advice. Senior people say don’t do this because they’ve seen it done and it’s an absolute nightmare (I’ve seen it happen multiple times).

It’s untyped, untestable, arcane, and unmaintabale. Even the link says don’t do this. Just don’t.

Congrats on having the chops (and neckbeard) to pull this off. If you want to do it in a personal project, more power to you. But it doesn’t scale at organizations.

Anyone writing code in a language they don't know will write nightmarish code. No matter if it is shell, Python, Go, or what have you. I have seen it happen multiple times as well, with all of these languages. And /that/'s the problem.

So, it depends on the context. "Organization" is a very broad term. If the team you are in is familiar with the shell, and the shell is the right tool for the job, there isn't that much of a problem.

I'm not arguing for writing large shell scripts when there are better alternatives. But, sometimes, it's the right choice and it can be written properly. And even if it isn't... 500 lines of poorly written shell aren't that many in any case, and they aren't that bad compared to 5000 lines of poorly written Python for example.

> Anyone writing code in a language they don't know will write nightmarish code. No matter if it is shell, Python, Go, or what have you. I have seen it happen multiple times as well, with all of these languages. And /that/'s the problem.

Came here to say this.

I fail to see why you think it would take more lines to do in Python.

And the problem with bash is similar to C: many people think they can write correct C code, but as it has been shown plenty of times, that doesn’t really work out in the end. Languages have different tradeoffs and bash really has no redeeming qualities, imo.

> I fail to see why you think it would take more lines to do in Python.

Because Bourne shell is not a Python-class language like Ruby or perhaps even Scheme or AWK are. Things that are easy in shell can be hard in Python and vice versa: awkward wrangling of subprocess.Popen in Python can be a single line of shell; simple tree munging with lxml in Python can easily be a hundred or more lines of wasteful xmlstarlet invocations in shell. Parallel processing, even without the heavy artillery of make or GNU parallel, is easy in shell but hard in Python. Structured data is hard in shell but easy in Python.

(Similarly, I wouldn’t use ML or Haskell to write a web scraper, and I wouldn’t use Python to write a compiler.)

Regarding parallel processing, I have to disagree. It is bad in shell, because it makes best practices such as "set -e" impractical. The usual failure mode is a script that does not stop correctly on SIGTERM, and its children have to be hunted down and killed separately.

That’s fair.

Perhaps a better way to put it would be that sloppy concurrency is easy. This doesn’t sound all that good, but it’s easy enough that I find myself parallelizing one-off things in shell that would be such a hassle to do nonsequentially in Python (compared to the magnitude of the problem) that the option wouldn’t even surface in my brain. (If I rewrite the shell script in Python later, the rewritten version is often sequential.)

The shell error handling story for “run stream processing steps in lockstep” parallelism is much better than for “spawn a bunch of identical children” parallelism, though, and that’s often enough.

I’ll bite. Show me the equivalent of this in python, including error handling, parallelism, disk footprint, and real-time status output:

   #!/bin/bash -eux -o pipefail

   TMPDIR=$(mktemp -d tempdirXXXXX)
   trap “rm -rf $TMPDIR” SIGINT SIGTERM ERR EXIT
   (
      cd $TMPDIR
      foo | grep x > foo.out & 
      bar > bar.out & 
      baz $TMP | tee baz.out &
   )
   wait

   tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh server.com bash -c “cat - > out.tag.zstd”

    $ shellcheck myscript
     
    Line 1:
    #!/bin/bash -eux -o pipefail
    ^-- SC2096 (error): On most OS, shebangs can only specify a single parameter.
     
    Line 4:
    trap "rm -rf $TMPDIR" SIGINT SIGTERM ERR EXIT
                 ^-- SC2064 (warning): Use single quotes, otherwise this expands now rather than when signalled.
     
    Line 6:
          cd $TMPDIR
             ^-- SC2086 (info): Double quote to prevent globbing and word splitting.

    Did you mean: (apply this, apply all SC2086)
          cd "$TMPDIR"
     
    Line 9:
          baz $TMP | tee baz.out &
              ^-- SC2086 (info): Double quote to prevent globbing and word splitting.

    Did you mean: (apply this, apply all SC2086)
          baz "$TMP" | tee baz.out &
     
    Line 13:
    tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh server.com bash -c "cat - > out.tag.zstd"
                 ^-- SC2086 (info): Double quote to prevent globbing and word splitting.
                 ^-- SC2153 (info): Possible misspelling: TEMPDIR may not be assigned. Did you mean TMPDIR?

    Did you mean: (apply this, apply all SC2086)
    tar -cf - -C "$TEMPDIR" | zstd -6 -z - | pv | ssh server.com bash -c "cat - > out.tag.zstd"

You can forego the final "bash -c" by leveraging dd(1) to avoid the redirection problem:

    ... | ssh server.com dd of=out.tag.zstd

Thanks! Will definitely use that moving forward.

dd(1) reads from STDIN by default unless given an "if=" parameter, and writes to STDOUT by default umless given an "of=" parameter.

Python wins the triathlon by being second best in every game. Piping things between programs is about the only thing bash is good at, nothing else, of course it is going to look good when unconditional straight piping things through standard programs is the only requirement. It's like saying c++ is fastest at calculating pi, duh, nobody will argue against that. Now, instead, put that script into a bigger context where some actual program logic or other integrations are needed. It is going to explode in unreadability, bugs and only supporting happy-cases. It is only simple if your original problem is simple.

I like both Bash and Python, so I think I'll give this a try. Should I implement the bugs as is, or fix those first?

You're implying that all languages are equally likely to produce buggy programs. That's clearly false. There's a spectrum.

At one end you have things like Haskell, OCaml, Rust, (or if you go really extreme Dafny etc.).

At the other end you have Bash.

There's no way a 500 line Bash script would require 5000 lines of Python.

Python is untyped and untestable. The only way I’ve seen people use it is by importing whatever libraries were popular in whatever year they wrote it, so it’s also arcane.

Also, to understand someone else’s python, you have to read at least 10x more lines of line noise, so it somehow manages to be less maintainable than shell.

Concrete example: Every python code base I’ve encountered has some function named exec that poorly reimplements shell job control and argument passing. As the python code bases grow, they end up with more than one of these, all of which have different bugs.

I usually discover this about 6 hours after the script has been sent to me, since someone who has heard the words “supply chain vulnerability” locked down whatever machine I need to run the script on.

The last ten times this has happened (at multiple companies, with multiple teams) it ended up being faster to figure out what shell commands the python script wanted to invoke, port a few things to jq and then reimplement the rest in a mix of shell and perl.

The result is always 1-10% the lines of code of the python, and actually runs without downloading an entire python distribution, dependency manager, and hundreds of broken packages at runtime.

>Python is untyped and untestable.

Completely untrue. You're conflating static typing with strong typing. And pytest is a great test framework.

And you're recommending shell for better typing and testability? The language where everything is a string, there are no proper arrays, and functions don't have local variables? You have to be trolling.

I’ve probably spent two weeks (80+ active hours) debugging stuff like ‘“” is not None’. At this point, I’d prefer a language with only string types to one that only checks types in production.

pytest is terrible, at least from how I’ve seen people use it. It hides the diagnostic error explaining why the test failed in a random place. Its output is not parseable. If you have a long running test that times out, then it buffers the stdout and stderr so you have to wait for the fallback test time out every. single. time.

You've got exactly same problem of undefined vs empty in sh (-n not being exact inverse of -z vs string comparision with != "", different behavior if -u is set, etc). Just check the stackoverflow answers on how to test if a bash string is empty or undefined and there isn't even consensus on what is the best way to do it.

Mypy has had strict Optional checking on as default since 2018, almost entirely eliminating any None-related errors, meaning it is actually stricter and safer than Java or other traditionally considered static languages in that regard, (bar any dynamic misuse in deserialization boundaries). Assuming you use mypy of course, but that's a given today imo, like using shellcheck is for sh.

On top of that, fn in Bash can only return their status code. Yes, command substitution exists but is full of footguns.

I cringe everytime my manager tells me not to write shell scripts but javascript instead. There is ShellCheck, Bats and there are extensions for e.g. Visual Studio Code or Jetbrains IDEA that make writing shell and bash scripts a pleasure. I find it generally much more productive to write a shell script because I already know all the gnu/unix userland tools. And i know this shell script will be portable and still running decades from now unless some version specific node script with dependencies.

> it is perfectly possible to write clean, maintainable, and testable shell scripts.

Possible? Maybe. Easy? No. Especially the “testable” part.

The vast majority of engineers do not know how to write clean shell, let alone testable shell. There’s a reason for this: it’s extremely difficult and non-obvious to do, and generally isn’t worth investing the time in learning (you can just use a different language instead: perhaps one with support for advanced features like… arrays…)

So sure, you can write your 500 line shell script and it may well end up being beautiful. But what happens in 5 years when someone who isn’t as enlightened as you has to maintain it?

(This all without mentioning the worst aspect of shell scripting: security. Good luck handling secrets without leaking them in one way or another).

> Possible? Maybe. Easy? No. Especially the “testable” part.

a testable shell script? Never seen one.

Thinking about scripts I've read in the past, I remember seeing Jason Donenfeld's bash script for wireguard-wg and thinking how productive and readable it was,

https://github.com/WireGuard/wireguard-tools/blob/master/src...

I see your point, but what happens in your alternate world of having to maintain the 5000-line Python version 5 years down the line? You face the same issues you describe. The people that wrote the original beautiful code aren't there anymore, and whoever comes touching the code "messes it up". Most Python "scripts" I have had to deal with that had been around for a while were pretty terrible as well.

First, I doubt that the SLOC conversion ratio between sh and Python is really as high as 10x.

Second, yes, I do think the 5000 line Python script would be more maintainable than than the 500 line sh script. I'd choose real types and data structures any day of the week.

Bad code is bad code, regardless of language. At least with Python, I more or less know all of the idioms and can at a glance understand what the flow of a program is trying to accomplish. Shell has a lot of esoteric voodoo syntax to replicate better string/path/array handling that is just a given in other languages.

I don’t think you do see my point: my point is that writing clean, testable, maintainable Python is both easier and a more widespread skill than writing equivalently clean, testable and maintainable shell. Both of these factors will mean that the Python remains maintainable for longer.

> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway.

This is an important distinction, and probably why I still make shell scripts.

I've found utilities like `shellcheck` invaluable for making robust scripts and maintaining them.

I love shellcheck. You can run it locally or via the web if you don't want to install it.

https://www.shellcheck.net/

The shell provides an interface to the filesystem and concurrent pipelines like no other readily-available language.

Many tasks are more concisely expressed and readable in shell than in any other language.

The fact that posix shell is somehow state of the art of concurrent pipeline and filesystem interface is really more a critique of state of the art than a praise of shell.

I'm doing programming language research and it's just infuriating how retrograde shell is. Its design is a fossil of history, and boy did our understanding of what makes a good language evolve. As a proof of this look at the jq tool. It is so powerful and clean. The implementation is simple and reasonably fast. It has been designed by a researcher from functional languages (an ocaml guy). This is a way to build a language around pipelining. At a lesser level, look at how moving from vimscript to lua in neovim has brought so many high-quality new plugins, look at how many people now tinker with low-level stuff and write high-perf libraries now that C/C++ is not the only choice. Surface language and having very little very orthogonal concepts is important.

Some core implementation patterns for PL are very old, but there have been lots of simplifications and rationalizations in surface language. Error handling, values hygiene, scoping, simple datastructures. Posix shell might have been a prowess in the 70s, now it is mostly a hack and this is a fact. Let's be a bit less superficial when honoring the unix mantras and dream about something better (making a better shell for its core ideas(!), that is, handling processes and fds).

The oil shell is an attempt to move forward:

https://www.oilshell.org/

Powershell is the new shell.

As it passes objects the filtering, sorting, joining is generic across files or any other data type.

And it is so wonderously mistaken in assigning a boolean type to $?.

I really cannot fathom how this could be, as Microsoft bundles curl, which has nearly 100 exit codes on various types of failure.

Whoever made this language decision needs to explain themselves.

> And it is so wonderously mistaken in assigning a boolean type to $?

Use $LASTEXITCODE for native executables, and $? for PowerShell cmdlets.

https://learn.microsoft.com/en-us/powershell/module/microsof...

Yep; shell isn't the best tool for every job, but the things it's good at, it's really good at.

Do you a good anecdote regarding shell scripts?

Long time ago some dude created a couple of korn shell scripts on Aix to automate file uploads and ETL related tasks.

With time that work moved from Aix into Red-Hat Linux, still korn shell, with the adaptations from Aix into GNU/Linux command line parameters and such.

I got tasked to port them into Java, because the team that got the job to maintain the servers lacked UNIX experience, and they thought selling the porting project was a good idea.

Feedback from customer, slower than those scripts and not as easy to change, what a surprise.

I use Korn shell (ksh93) for quick scripts. Besides arrays and such (it's approximately a superset of bash, since bash copied some but not all of ksh), the killer feature is a declarative extended `getopts` that makes it trivial to provide even the quickest and dirtiest script with short/long option processing with full help, so that the purpose and usage of the script remain comprehensible five days or five years later. https://gist.github.com/kpschoedel/91fdcfa934111f3846efb5034...

> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway.

Agreed on avoiding rewrites if possible but 1/10 ratio is laughable.

> But if one approaches them thinking that “the shell is a real language and must be written as such”, with solid principles in mind

Then one would simply choose a proper language that had more of those solid principles laid at its foundation

Unix shell is a proper language and has very solid principles beautifully laid at its foundation by Douglas McIlroy.

I understood that Stephen Borne's love of ALGOL was the chief structural influence of the POSIX shell (so much so that he used the C preprocessor to turn his source code into fake ALGOL).

"Stephen Bourne's coding style was influenced by his experience with the ALGOL 68C compiler that he had been working on at Cambridge University... Moreover, – although the v7 shell is written in C – Bourne took advantage of some macros to give the C source code an ALGOL 68 flavor. These macros (along with the finger command distributed in Unix version 4.2BSD) inspired the International Obfuscated C Code Contest (IOCCC)."

https://en.m.wikipedia.org/wiki/Bourne_shell

Do you have some writing on this? I'd love to read it.

The question is if the script outgrew its original intended lifespan.

In my opinion, shell scripts should be used either as glue or due to portability reasons. Rewriting a perfectly good and working script just because makes no sense.

So if the script started as some quick and easy way to deploy to prod but slowly becomes a massive cli "one tool to rule them all" kind of thing.. maybe it's time to change course.

The reason I don’t like shell scripts are spaces and the weird constructs sometimes.

I never got into Perl, but I understand why it was popular.

Sure it’s possible, but it’s therefore also brittle, when someone touches it and is not a complete expert.

Shell is nice not because of the shell language, but the ease of executing and chaining other programs.

There are even great unit testing frameworks for testing across a variety of shells and versions. This can be integrated pretty easily into existing build tools and run the same way the other code it may (or may not) support is run.

shell is loose and offers a lot of power per line, so it’s pretty easy to make a spaghetti nightmare. but it’s very possible to write nice shell.

i dunno about testing it, i feel like that’s getting pretty close to the barrier of shell’s utility

A program in any language can be maintainable when written properly and maintained by someone who is capable, diligent, and has a good test setup.

Bash is turing-complete, so it's possible to write automated test cases in bash. Example: https://github.com/vlisivka/bash-modules/blob/master/bash-mo...

Reply on news.ycombinator.com