ShellCheck: Finds bugs in your shell scripts | Hacker News Reader

ShellCheck: Finds bugs in your shell scripts

shellcheck.net

301 pointssuperasn54 comments

Discussed a few weeks ago: ShellCheck: A static analysis tool for shell scripts - https://news.ycombinator.com/item?id=26504661 - March 2021 (73 comments)

(We mark stories as dupes if they've had significant attention in the last year or so).

Previous threads, for those who are curious:

Lessons learned from writing ShellCheck - https://news.ycombinator.com/item?id=22279585 - Feb 2020 (46 comments)

Shellcheck: a static analysis tool for shell scripts - https://news.ycombinator.com/item?id=9001931 - Feb 2015 (46 comments)

ShellCheck: a static analysis and linting tool for sh/bash scripts - https://news.ycombinator.com/item?id=8777705 - Dec 2014 (20 comments)

ShellCheck – Online shell script analyzer - https://news.ycombinator.com/item?id=8182745 - Aug 2014 (14 comments)

Oops sorry about that. Just found out about this wonderful program from this HN comment here(1). Should have been more careful to check it through algolia before posting though.

(1) https://news.ycombinator.com/item?id=16085914

Not to worry! It happens.

Some other shell script check and formatting tools:

All shells take the -n option to perform a basic syntax check.

Debian has a script for checking scripts for bashisms:

https://manpages.debian.org/checkbashisms

bashate is a automated style checker for bash (similar to pep8 for python):

https://opendev.org/openstack/bashate

lintshell is an early prototype of a shell linter based on the Morbig trustworthy static parser for POSIX shell, based on the Why3 platform for deductive program verification.

https://www.irif.fr/~treinen/colis/ https://github.com/colis-anr/morbig https://github.com/colis-anr/lintshell

Shellharden helps rewrite scripts for ShellCheck compliance:

https://github.com/anordal/shellharden/

shfmt is gofmt for shell:

https://github.com/mvdan/sh

ShellScriptFormatter is another formatter:

https://github.com/osalvador/ShellScriptFormatter

Some of these tools and other tools can be automatically run by check-all-the-things:

https://github.com/collab-qa/check-all-the-things/

The best best part of Shellcheck is that the errors it produces have a short code that can be looked up in their wiki. The wiki explains what you did, why it’s bad, and how to fix it. Using Shellcheck teaches you to write better Bash, rather than just yelling at you with inscrutable linter violations.

If you use iTerm2 (or I think this feature is available in other terminal software) you can use "Smart Selections" to transform any ShellCheck errors into links to the wiki.

1. Create a Smart Selection with the regular expression "SC\d+"

2. Add an action to "Open URL...", with the URL "https://github.com/koalaman/shellcheck/wiki/\0" (\0 will be replaced with the matched regex)

3. Now holding down the command key will change any ShellCheck errors to be active http links.

BASH is not an appropriate standard to use in writing a script.

ShellCheck recognizes three hashbangs, POSIX, BASH, and the Korn shell (as is explained in the manual page). I would argue and assert that BASH is the least relevant of the three.

I have occasionally needed to write to the POSIX standard, and remove advanced features from my scripts. This is most commonly needed for BusyBox (including Windows) and Ubuntu/Debian DASH. It is important to understand what is not in POSIX, for the time when an advanced shell isn't available. Please browse the POSIX definition, reported by a google of "POSIX shell":

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...

The Korn shell lived and diverged for many years, beginning with ksh88 (issued before the first BASH release in 1989). David Korn retired, and AT&T was not pleased with ksh93 changes implemented by outsiders who impacted both compatibility and performance, and thus ksh93 was rewound.

Cygwin, notibly, only implements mksh for Windows. The mksh binary on all platforms is a fraction of the size of BASH (check the output of the "size" command if an "ls" does not persuade you).

However, the MirBSD Korn shell is one of the most lively shells for development activity, the latest release being in October of 2020. It is also on every Android phone (chosen as the system shell by Google), so its deployment is massive.

When I am purposely writing above the POSIX shell, I choose mksh from MirBSD as my standard. I do not think the BASH innovations to be well-thought.

The authors of Bash don’t capitalize Bash

If only they had chosen a more reasonable improvement on coprocesses, I would have used whatever case they wanted.

> However, the MirBSD Korn shell is one of the most lively shells for development activity, the latest release being in October of 2020.

https://lists.gnu.org/archive/html/info-gnu/2020-12/msg00003...

As said, far more active.

bash 5.0: https://www.osnews.com/story/129062/bash-5-0-released/ 2019-01-08

MirBSD Korn 2020:

    Oct 31, 2020 mksh-R59c
    May 16, 2020 mksh-R59b
    Apr 14, 2020 mksh-R59
    Mar 27, 2020 mksh-R58
    Mar 1, 2019 mksh-R57

As said:

> the MirBSD Korn shell is one of the most lively shells for development activity, the latest release being in October of 2020

I.e., you suggested the measure is release recency, not cadence. And after I pointed out that bash (sorry, BASH) has a more-recent release, you're moving the goalposts.

Anyways. It doesn't matter. Love the shell(s) you love; hate the shell(s) you hate. You don't need a quantitative basis. They can release as often or as rarely as they like with little impact on whether they're good shells or not.

Also: https://github.com/oilshell/oil/branches/all?query=release%2...

As much of the scripting that I am required to maintain emerged from HP-UX, the "POSIX" shell on my legacy platform emerged from Korn (and I have the manual page to prove it - the text of the HP-UX "POSIX" shell manual page includes arrays and a number of other Korn features).

The reason that the POSIX shell is not Korn is the existence of Microsoft Xenix, where the maximum size of the code/text segment is 64k.

Debian chose the POSIX standard for precisely this reason: a minimal text segment.

One particular thing that I like about Korn is this:

    function getv { nameref var="$1"; shift; print "$@"; read var; }

This cannot be done with (my versions of) "bash." ShellCheck flips a gasket over this (legal) syntax.

I will post some of my scripting that uses this feature if you have interest.

> However, the MirBSD Korn shell is one of the most lively shells for development activity, the latest release being in October of 2020.

The OpenBSD Korn shell is actively developed and there is a portable version of it available. The latest version which reflects the shell as available in OpenBSD 6.9 was released 3 days ago. https://github.com/ibara/oksh

I prefer the MirBSD variant, as does Google.

This is one question upon which we agree.

> ShellCheck recognizes three hashbangs, POSIX, BASH, and the Korn shell (as is explained in the manual page). I would argue and assert that BASH is the least relevant of the three.

Depends on how you call "relevant", but I disagree regardless. If you're writing a shell script, you either need to be hard-core portable, in which case POSIX is your only option, or you don't, in which case you're almost certainly targeting a GNU/Linux system with BASH installed. Of course, if you know that you're targeting a specific system with a specific shell installed then by all means; if I were primarily an OpenBSD dev, I'd happily use ksh. But in general, the (unix) world is always going to give you POSIX, and BASH is the second most widely available option. And bluntly, for all that I love shell, if you're trying to argue for the most elegant or sophisticated language, sh/BASH/ksh is irrelevant because all of them kinda suck outside of their niche.

Oracle requires ksh93 when installing their database.

For myself, I write to the ksh93 standard, but reduced to mksh, because I desire functionality, but portability.

The question of s/BASH/bash is not relevant to me. The toolset is the question.

Noted again is that Cygwin only implements mksh, not ksh93. Write for what you have.

It would make me nothing but happy should "bash" adopt ksh syntax. Perhaps I should struggle to be sufficiently adept to submit patches.

Oracle's list of "required packages" is annoyingly large but I cannot find a particular shell amongst them. The installation guide appears to have a paragraph devoted to each of bash, "bourne or korn", and C shells.

You also do not realize that my Oracle strobe script runs on Windows, with the cygwin mksh binary.

I wrote it that way.

Anyone who thinks that "bash" is modern should read my POSIX critique.

https://news.ycombinator.com/item?id=26178257

I also did not think anything but mksh to be appropriate for handling inotify.

https://www.linuxjournal.com/content/linux-filesystem-events...

What many of you do not realize is the writing of POSIX scripts, for Windows, and how this is done.

This is the most direct method.

https://frippery.org/busybox/

They should add the wiki URLs as links to those error codes then (if supported by the terminal) - GCC has done that recently for the -W flags shown in warning or error messages.

Some related resources, for intermediate shell users already familiar with shellcheck and looking to dig deeper into the history underlying obscure corner cases, include the developer Vidar's blog (@ https://www.vidarholen.net/contents/blog/), the blog of the Oil-Shell developer (@ https://www.oilshell.org/blog/), and Greg's Wiki (@ https://mywiki.wooledge.org/BashPitfalls).

This is one of the absolutely essential tools for anyone who writes a non-trivial amount of shell scripts.

Maybe it says something about the people I work with, but I've never seen a fully valid shell script, over 20 lines long, that handled spaces in paths and command errors properly, that didn't first go through ShellCheck. I've noticed that hardcore *nix people are usually some of the worst offenders, with their phobia of spaces, from years of broken shell scripts, completely preventing them from even thinking about the possibility of a space in a path, with some considering the space the bug, rather than its handling.

I've found that the best way to introduce someone to ShellCheck is to run it on their script, right in front of them, with most saying something like "How did I not know about this?". Linters are great!

The kind of issues shellcheck catches are mostly of the "didn't learn shell (and didn't believe that shell was a language that had to be learned)" kind. Or to put it another way, once you have a good hang of the shell language's peculiarities (which differ from most general-purpose languages, but are fairly self-consistent with a few high-frequency exceptions) ... well, less of your bugs are ones that shellcheck can catch.

I consider it an example of the general trend against learning domain-specific languages.

I can and do write shell scripts all the time which properly handle spaces, without ShellCheck. Not to say that it isn’t good to use, especially in a professional environment, but it’s not that hard. For parallel processing, xargs -0 and find -0 are your friends!

Oh good heavens, my friend.

My boss and coworker just spent the weekend on seven million files on my script.

find . -type f -print0 | xargs -0 (cygwin) sed 's/\ff//'

They think me a genius. This is nothing.

Did your regex get munged there? \f (form feed) is something I've never come across (well maybe back in BBC basic?) - what's the context of removing it/them (I'm guessing only pairs \f\f) from 7M files?

I have.

There are a few simple rules that, once you learn them, eliminate those kinds of problems. The main one is always use double-quotes around variable dereferences. The second is learning a few patterns, like find ... -print0 | xargs -0 ...

But even after you've learned those simple rules, shellcheck is still useful. It will find fewer problems, but that's a good thing, that means you learned something :-).

Agreed. I have it built into my vim editor, running after saves. Not only does it help catch errors, it catches potential errors, poor formatting and has become an amazing learning tool. Bash isn’t my favorite language, but many of the issues I had with it are no longer relevant.

There is a convenient VS Code plugin for ShellCheck. https://marketplace.visualstudio.com/items?itemName=timonwon...

ShellCheck and templates from the Shell Field Guide[1] is what keeps my bank running. Rust cmd_lib[2] is tempting with safety, though tougher maintainability.

[1] https://raimonster.com/scripting-field-guide/

[2] https://github.com/rust-shell-script/rust_cmd_lib

I love ShellCheck. It’s good at providing hints at improving your style too.

probably one of the more widely-deployed Haskell programs

This and `pandoc` are solid proof that Haskell is a superb language for processing structured text (be it for compilation, or translation, or checking).

Or that people who are good at processing structured text tend to use/like Haskell.

(Not disagreeing with you, but the causal direction isn't immediately obvious to me.)

I do admire it, for it's grieovus errors.

It does not understand "nameref" amongst other flaws.

Unfortunately, last I checked, shellcheck is not compatible with M1 macs yet. I love the tool so much that I while I wait for compatibility, I manually check my shell scripts at the web UI.

Yeah, there are still issues with Haskell and the new Mac architecture. Pandoc and ShellCheck are two of the essential tools affected. In the case of ShellCheck, I just downloaded an x86 binary from the GitHub Releases page and put it in /usr/local/bin. Good enough for the time being.

https://github.com/koalaman/shellcheck/releases

ShellCheck is a great tool! If you're looking for an easy way to run static analysis on .sh files continuously, you might want to look at DeepSource. ShellCheck is supported. https://deepsource.io/gh/deepsourcelabs/demo-shell/issues/

One of the biggest value of shellcheck is that all of its errors are documented on their wiki, and easily searchable. If you don't understand why there is an error, the wiki page will tell you why an idiom is bad, and in some cases point you to examples of how it might fail, and you'll learn something new.

I wish the wiki was packaged into the tool for offline consumption.

It found several things in my script that I’ve done wrong for over 20 years...

ShellCheck is absolutely essential. Make sure to get the vscode plugin as well. I love it.

The error code can be googled and it the first result will be exactly what you need. One of the best projects to happen to open source.

I’ve written some pretty horrible Bash applications such as a load testing tool for a blockchain, probably hitting > 2kloc. ShellCheck and ShUnit made that possible.

This is a very helpful tool, I can only recommend it. Works perfectly well with vim and Syntastic.

> E044: Use [[ for =~,<,> comparisions

Is there any reason to prefer bashisms over POSIX?

That is bashate, not ShellCheck, but there is good reason to use [[ since these operators are not POSIX to begin with and don't work in [.

because its 2021 and we should get over compatibility with archaic tech?

In Debian/Ubuntu /bin/sh, most certainly.

Because the =~ and > and < operators aren't defined for [ in POSIX.

Also ShellCheck suggests similar improvement only when your shebang says that bash is the expected interpreter instead of sh (or other ways of denoting POSIX compat).

The first plugin i install in Intellij alongside IdeaVim.

I can highly recommend this tool. A few years ago, I ran it over our shell scripts and it gave us many good recommendations and even found some bugs. I learned a lot about shell scripting during that time. Really, I cannot recommend it enough, what a fantastic tool.

Reply on news.ycombinator.com