ShellCheck: A static analysis tool for shell scripts | Hacker News Reader

ShellCheck: A static analysis tool for shell scripts

github.com

399 pointsducktective73 comments

If curious, the interesting past threads seem to be:

Lessons learned from writing ShellCheck - https://news.ycombinator.com/item?id=22279585 - Feb 2020 (46 comments)

Shellcheck: a static analysis tool for shell scripts - https://news.ycombinator.com/item?id=9001931 - Feb 2015 (46 comments)

ShellCheck: a static analysis and linting tool for sh/bash scripts - https://news.ycombinator.com/item?id=8777705 - Dec 2014 (20 comments)

ShellCheck – Online shell script analyzer - https://news.ycombinator.com/item?id=8182745 - Aug 2014 (14 comments)

I much prefer this to the format you were using several months ago :)

One of my favorite things about Shellcheck is that the error messages come with a link to a wiki page about the problem. Does anyone know other programs that have a similar approach to error messages?

https://github.com/koalaman/shellcheck/wiki/Checks

Rust Compiler Error Index

https://doc.rust-lang.org/error-index.html

Related, Clippy, a linter for Rust also does the same and links to https://rust-lang.github.io/rust-clippy/master/index.html .

Yeah, C# compilers do. Easily Googleable error code to further explain it.

I think it's a Microsoft thing. A ton of their stuff has unique error codes just for this purpose.

Remember back when OS/2 had a special error code for every error message?

Codes like that are a part of IBM culture that spilled out to other companies.

Sure beats the General Protection Fault we were so accustomed to back in the day. :)

they had the MSDN cd (microsoft developer network) for documentation in these days; that cd had a lot of thorough documentation for apis and developer tools.

All heavily internationalized software should have error codes.

PHP was just ahead of the T_PAAMAYIM_NEKUDOTAYIM curve.

I recall working with AngularJS, which output error messages that had links, which opened the documentation with bits from your stack trace filled in.

It was a while ago and seemed a bit gimmicky, but i really enjoyed the ability to see what could be done about an error by opening a web page (without having to manually dig through StackOverflow or other sites).

Now, if we'd get a development framework that'd couple documentation like that, with user generated comments (like PHP has, for example see the bottom of https://www.php.net/manual/en/function.strcmp.php ), then i think the developer experience would improve a bunch!

Who knows, maybe even allow users to contribute possible fixes and allow those as autocomplete solutions and you've just improved on what tools like Codota do a whole bunch!

Hadolint is another. It's built atop shellcheck.

https://github.com/hadolint/hadolint

find-sec-bugs does that. It's used by, for example, SonarQube.

See https://github.com/find-sec-bugs/find-sec-bugs/blob/master/f...

Error Prone does this: https://errorprone.info/bugpatterns

Errorprone (java compiler plug-in) has this as does NullAway (an errorprone plug-in)

I agree that it’s really handy, especially when implementing the check in a CI pipeline with lots of developers.

Many of the JS linters do as well (eslint, etc).

SQLAlchemy also does something similar. It points to relevant SQLAlchemy page for more information about the error.

Dart's static analyzer & lint rules: https://dart-lang.github.io/linter/lints/

Each rule has its own url. Tool therefore easily gives you a link.

If you're building a REST API, make sure to check out RFC7807 ( https://tools.ietf.org/html/rfc7807 ) , which recommends doing exactly this. Giving each error a type, which is a resolvable URI.

zpool status

Newer versions of GCC have clickable (using terminal escape sequences) warning flags in the output that link to the corresponding documentation.

Staticcheck for Go kind of does something similar in that it outputs Googleable error codes, but doesn't actually print a link to the pages. It's also included as part of golangci-lint.

https://staticcheck.io/docs/checks

This is one of my favorite tips for almost effortlessly improving devops productivity (along with using https://github.com/mvdan/sh for auto-formatting) — by now shellcheck has good editor support in e.g. VSCode and I have a standard pre-commit.com hook for all of my projects. I've seen so many long iterations where people flail at a complex shell script for something like a cron job which was immediately flagged by shellcheck. I generally recommend rewriting in Python but this is a good less invasive step.

One thing which might be worth considering is adding this to your personal ~/.shellcheckrc to make it more pedantic:

enable=all

I went looking through the issue tracker to see if there's an issue about supporting $XDG_CONFIG_HOME and was pleasantly surprised to find out that it seems to already work. So ~/.config/shellcheckrc works just as well as ~/.shellcheckrc

ShellCheck made me confident enough to write more complex bash scripts. Highly recommend! Works great with emacs + flycheck.

It is also a quite nice tool for checking the POSIX compatibility of shell scripts. I find it very useful when working on Solaris or some other UNIX that does not use Bash by default.

Same here, I basically learned how to write POSIX-compliant Shell with the help of ShellCheck. Shell is not a difficult language to learn but it definitely has its oddities and the different flavours (Bash etc.) only add to the confusion.

What people don't always expect is the immense portability that POSIX-compliant Shell offers you. This thing runs on pretty much everything.

Another tool for that is checkbashisms from the Debian devscripts package:

https://manpages.debian.org/checkbashisms

haskell forbidding people to leave fragile bash to learn babashka !

Great tool that everyone should be using for shell scripts!

Then again, I've learnt more than once that you start with "I'm just capturing a few commands in a script" and the next thing you see is a mess of special characters, crude syntax and nasty error handling. Don't be like me...

I find it helps if you have a hard rewrite rule. I really love the convenience of shell scripts for small scripts, but as soon as I have one that crosses 100 lines I rewrite it in python. The python version is a bit less convenient for spawning processes and piping them together, but it's so much more maintainable that you can scale it up to quite a bit larger without as much worry. 100 lines is a nice size where it doesn't take very long to rewrite, so even though I figure I could get away with a longer shell script without it getting too bad, I'm definitely not going to want to rewrite it once it gets big enough to be a problem.

Invoke [1] can make it a bit more convenient.

[1] https://www.pyinvoke.org/

I am you, except my go-to is Ruby.

And then you get to google scale and realize using python for every config, like your entire build system, leads to a huge mess of completely non-deterministic build scripts that do wacky things like make HTTP calls to prod systems, depend on all kinds of global state stores, etc. So you build a new version of python that strips out all of the non-determinism and move your config to that (see starlark: https://docs.bazel.build/versions/master/skylark/language.ht... ).

I know I'm in the minority but I just don't care for python for sysadmin / devops type tasks. It fails in weird ways on file and directory manipulations and I just don't find it's stack traces very informative. A lot of scripts seem to be a bunch of system () calls strung together, how is that better than well written Shell code?

What I love python for is scraping web pages and dealing with JSON or xml it rocks for that.

Some other shell script check and formatting tools:

All shells take the -n option to perform a basic syntax check.

Debian has a script for checking scripts for bashisms:

https://manpages.debian.org/checkbashisms

bashate is a automated style checker for bash (similar to pep8 for python):

https://opendev.org/openstack/bashate

lintshell is an early prototype of a shell linter based on the Morbig trustworthy static parser for POSIX shell, based on the Why3 platform for deductive program verification.

https://www.irif.fr/~treinen/colis/ https://github.com/colis-anr/morbig https://github.com/colis-anr/lintshell

Shellharden helps rewrite scripts for ShellCheck compliance:

https://github.com/anordal/shellharden/

shfmt is gofmt for shell:

https://github.com/mvdan/sh

ShellScriptFormatter is another formatter:

https://github.com/osalvador/ShellScriptFormatter

Some of these tools and other tools can be automatically run by check-all-the-things:

https://github.com/collab-qa/check-all-the-things/

Quality links there. Thanks!

To use in combination with shfmt : https://github.com/mvdan/sh

I really like task (http://taskfile.dev/) as a more usable layer on top of mydan/sh. Task is basically make but with a yaml syntax and built-in posix shell interpreter all in a single statically linked executable. It makes building something even on Windows a total breeze.

This has been brought up ten times since 2014 when it first appears on HN :)

Click on the site link[1] in parenthesis to the right of the title, you will see.

[1]: https://news.ycombinator.com/from?site=github.com/koalaman

It is actually good enough to be here that often. More often, even.

It's like in this xkcd. It's always new to somebody.

https://xkcd.com/1053/

And 23 more for https://news.ycombinator.com/from?site=shellcheck.net :)

Not once did I see it till now, its not like people are here 365 days or new people dont join.

Since the project I work on (https://spdk.io) largely produces a set of executables as output, it was most natural to write the tests in bash. There's one top level bash script that kicks off the full suite of tests and thousands and thousands of lines of tests all written as bash scripts stringing together calls to these executables.

One of these tests is to run shellcheck against all of the scripts in the repo. We don't allow any modifications to scripts without shellcheck giving them the green light now. The quality of our tests has increased dramatically since this was instituted - it's a really great tool.

Can you expand on why y'all thought Bash was a natural fit to kick off all your tests?

And talking about tests... I recommend [bats](https://github.com/bats-core/bats-core) for testing! I'm slowly adding tests to my dotfiles using this.

This is one of the best pieces of software. It's really changed the game and I don't think it would be easy to safely use shell in large teams without it.

I love linters and formatters. Often languages carry baggage and other “bad parts” that linters help you avoid.

For someone who’s not used to them they might feel annoying (e.g. why should I wrap every $() in quotes?! Ugh) but once you accept them your code will be a million times better and safer.

Same. Few years ago I started a little list of linters. Just kept adding tools and the list is huge now thanks to lots of contributors. https://github.com/analysis-tools-dev/static-analysis Still love adding new linters there when I cross their path. :)

thanks! you also have https://github.com/analysis-tools-dev/dynamic-analysis for stuff like valgrind; A star from me for both of them.

I believe they are near-essential for properly learning a language.

I can credit Resharper for a large part of my proficiency in C# for example.

A great highlight of how great Haskell parser combinators can be in the domain of linters.

Not near a terminal right now but it would be fun to run my shell history through this to see if I learn anything. Anyone try it?

Unless you're composing very rich commands, likely not much will show up. It exceeds at finding things like interpolation gotchas, bad loop control, etc. Much more structural than just one off commands (though would likely bug about a bunch of useless `cat`s when you can use redirect operators instead (i.e. `cat foo.txt | grep` is an antipattern according to shellcheck)

shellcheck-repl: Validation of Shell Commands Before Evaluation

https://github.com/HenrikBengtsson/shellcheck-repl

This tool validates your commands at the Bash prompt using ShellCheck and refuses to evaluate them if there's a mistake. It ignores a set of rules that doesn't play well with oneliners.

(Disclaimer: I'm one of the authors)

Alongside ShellCheck, I also use shUnit2 as my unit testing framework. These two combined have let me write some pretty long (~1k loc) bash programs that work. Yes, you should test your bash.

https://github.com/kward/shunit2

There are quite a few unit-testing frameworks for bash. I wonder, have you reviewed and compared them and came to `shunit2`?

I tried one or two but decided on shUnit2 because it seemed to be the most friendly and familiar. If you have suggestions of alternatives I'd love to hear them!

I haven't used it, but I have seen something similar called shellharden. It pitches it's advantage as being able to auto apply the changes.

https://github.com/anordal/shellharden

I use ShellCheck, its a great linting tool. All the vim linting frontends support it.

yeah, for integration with coc-nvim, take a look at this github comment: https://github.com/meatwallace/dotfiles/issues/129#issuecomm...

There is also a LSP at : https://github.com/bash-lsp/bash-language-server

There is a plugin for intellij based IDEs. The first thing i install on a new installation alongside IdeaVim (which is the best Vim emulation IMHO)

Can confirm, ShellCheck is awesome.

I just tried to use it this morning. I had an if with [[ ]] and one with [ ] so I hoped for advice. No errors or warnings. Then I didn't bother using it anymore. Furthermore, I switched the script to use deno!

shellcheck won't force you to prefer one of the two variants, hence the difference is relatively subtle to a degree that even documentation is not superhelpful either: https://www.oilshell.org/blog/2016/10/12.html

shellcheck will show you many more mistakes that are much more painful your example and also very common.

Deno is definitively an interesting language, but Shell scripting is the gateway to bootstrap so many of your beloved programming languages. If your environment allows it always prefer a proper programming language over shell scripting. Shell scripts are meant for Unix operating system operations, setting few variables, combining certain data-sources, doing filesystem operations, ...

A useful thing with conditionals is shellcheck will warn you not to write [ foo -eq 1 && bar -eq 2 ], but to write [ foo -eq 1 ] && [ bar -eq 2 ] or [[ foo -eq 1 && bar -eq 2 ]].

I do think it would be nice if shellcheck was more opinionated, or if it was possible to create rulesets that have stronger opinions. For example, I would personally like to enforce the use of double bracket conditionals in my scripts, but I know that others prefer the single bracket variety.

What would be better is that undesirable syntax support is aggressively deprecated and ultimately removed.

Which shebang was your script using?

If it was

  #!/usr/bin/env sh

then shellheck would report

  SC3010: In POSIX sh, [[ ]] is undefined.

.

It was with bash, not sh. Hm interesting.

Those are both valid in bash. If you were using posix sh it would error.

Reply on news.ycombinator.com