Shellcheck finds bugs in your shell scripts
shellcheck.net
shellcheck.net
- It's almost always preferable to put `-u` (nounset) in your shebang to cause using undeclared variables to be an error. The only exception I typically run across is expansion of arrays using "${arr[@]}" syntax -- if the array is empty, this is considered unbound.
- You can use `-n` (noexec) as a poor-man's attempt at a dry-run, as this will prevent execution of commands.
- Also handy is `-e` (errexit), but you must take care to observe that essentially, this only causes "naked" commands that fail to cause an exit. Personally, I prefer to avoid this and append `|| fail "..."` to commands liberally.
Any particular reason why in the shebang instead of set -u?
> The only exception I typically run across is expansion of arrays using "${arr[@]}" syntax
In Bash? Works for me. Edit: another comment mentions it as well. Seem to behave better in newer versions of Bash and only problematic in <= 4.3 https://news.ycombinator.com/item?id=38397241
$ bash -uc 'unset x; echo "=> ${x[@]}"'
=>
$ bash -uc 'x=(); echo "=> ${x[@]}"'
=>
$ bash -uc 'x=(); echo "=> ${x[0]}"'
bash: x[0]: unbound variable
$ bash -uc 'unset x; echo "=> ${x[@]:-null}"'
=> null
It is unfortunately very confusion with functions. Made me like it less over the years.
The clarifications on `-u` and arrays are useful. I'm definitely used to assuming newer (... non-ancient?) versions of Bash are what is available.
In any case, you can workaround expanding empty arrays throwing unbound by using the ${var+alter} expansion
echo "${arr+${arr[@]}}" The problem with "${arr[@]}" only exists on bash 3 and before, since bash 4
$ bash --version
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)
$ declare -A arr
$ set -u
$ "${arr[@]}"
-bash: arr[@]: unbound variablemacOS is the only one out there missing on this. The other big feature that was only added after bash 3 and is missing on mac is associative arrays
GPL3, buddy. You can use brew to install a newer version and somewhat hide the old one.
"trap" is a great scripting feature that doesn't get talked about enough, IME.
Using `set` (e.g. `set -euo pipefail` as the first line after the shebang) is a common suggestion due to this, and similarly it helps with the `#!/usr/bin/env bash` case where the shebang can't handle additional arguments :-)
Unfortunately, shellcheck did not catch that. At least not with the default settings. But if you are implementing anything remotely security-critical, you should not be using shell anyway.
He waited a bit, and promptly said to never do that and started to explain the security risks.
https://github.com/jamespwilliams/strictbash, I wrote this little wrapper a while back that you can use as a shebang for scripts. It runs shellcheck for you before the script executes, so it’s not possible to run the script at all if there are failures. It also sets all the bash “strict mode” [0] flags.
[0] http://redsymbol.net/articles/unofficial-bash-strict-mode/
* https://news.ycombinator.com/from?site=shellcheck.net
with the last large-scale discussion (301 points; 54 comments) being in 2021:
Afaik a Turtle program has a single current directory, which makes it hard when you want to run concurrent jobs that need to be executed from particular directories. I partially solved the problem by using locks/queues/workers. But it got too much for me when Turtle started failing due to its current directory being deleted.
In contrast, typed-process lets you spawn separate processes, and execute within a working dir (rather than needing to cd there), so it works great for big, complicated workflows.
And it also has good support for OverloadedStrings, which means you can generally copy & paste what you would have typed into bash, and it just works.
I also use the interpolate package (with QuasiQuotes) to make the raw strings nicer in the source code, but it's not compatible with hlint, so I'm thinking of looking for a different package for string-handling.
I ended up liking PyF [1] for quasiquoting. A friend of mine worked with the author a while ago to strip down it's dependencies, so it's easy to justify. It's what I recommend in my cargo culting guide [2].
[0] https://hackage.haskell.org/package/shh
[1] https://hackage.haskell.org/package/PyF
[2] https://github.com/luke-clifton/shh/blob/master/docs/porting...
I very much live by the "fix all warnings before you commit" (or at least before you merge), so I have Shellcheck and a bunch of other linters set up in my pre-commit configurations. But the majority of the shell in most of my projects ends up embedded in .gitlab-ci.yml files, where it's hard to check. So I made a wrapper that does that automatically: https://pypi.org/project/glscpc/.
It uses the Shellcheck project and some magic to give you the Shellcheck remarks with (mostly) accurate line-numbers.
I don’t use GitLab CI, but I do use a good handful of other file types that essentially inline shell scripts. Dockerfiles, GitHub Actions, and Justfiles, just to name a couple.
Usually, and almost exclusively for the sake of ShellCheck, I make a point of putting anything more complex than a couple of commands into their own shell scripts that I call from the inlined script in my Dockerfile.
(This pattern also helps me keep my CI from being too locked into GitHub Actions)
It’s really not necessary to support inline except for single lines that are very short (under 30 chars).
Setting something like a 30 char limit will inevitably lead to code-golfing to fit a mess into the 30 char limit.
But that becomes harder to automatically check, that's why you should still have good peer reviews based on written standards somewhere.
No more linear Dockerfile, use the powers of your preferred language
We're very active in our Discord server if you have any questions! https://discord.gg/invite/dagger-io
Dagger has one very big downside IMO: It does not have native integration with Gitlab, so you end up having to use Docker-in-Docker and just running dagger as a job in your pipeline.
This causes a number of issues:
It clumps all your previously separated steps into a single step in the Gitlab pipeline. That doesn't matter too much for console output (although it does when your different steps should run on different runners), but is very annoying if you use Gitlab CI's built in parsing of junit/coverage/... files, since you now have extra layers of context to dig trough when tests fail etc. Plus not all of these allow for multiple input files, so now you have to add extra merging steps.
If your job already uses Docker-in-Docker for something, you have to be careful not to end up with Docker-in-Docker-in-Docker situations, or container name conflicts if you just pass through the DOCKER_HOST variable.
The one thing that would make this worth it is being able to run the pipelines locally to debug, but I've just written quick-and-dirty scripts to do that every time I've needed it. For example, running the test job in our pipeline on every Python version: https://gitlab.com/Qteal/oss/gitlabci-shellcheck-precommit/-...
The purpose of Dagger it's not to replace your entire CI (Gitlab in your case). As you can see from our website (https://dagger.io/engine), it works and integrates with all the current CI providers. Where Dagger really shines is to help you and your teams move all the artisanal scripts encoded in YAML into actual code and run them in containers through a fluent SDK which can be written in your language of choice. This unlocks a lot of benefits which are detailed in our docs (https://docs.dagger.io/).
> Dagger has one very big downside IMO: It does not have native integration with Gitlab, so you end up having to use Docker-in-Docker and just running dagger as a job in your pipeline.
Dagger doesn't depend on Docker. We're just conveniently using Docker (and other container runtimes) as it's generally available pretty much everywhere by default as a way to bootstrap the Dagger Engine. You can read more about the Dagger architecture here: https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-o...
As you can see from our docs (https://docs.dagger.io/759201/gitlab-google-cloud/#step-5-cr...), we're leveraging the *default* Gitlab CI `docker` service to bootstrap the engine. There's no `docker-in-docker` happening there.
> It clumps all your previously separated steps into a single step in the Gitlab pipeline.
It's not generally how we recommend to start, we should definitely improve our docs to reflect that. You can organize your dagger pipelines in multiple functions and call them in separate Gitlab jobs as you're currently doing. For example, you can do the following:
```.gitlab-ci.yml
build:
script:
# no funky untestable shellscripts here, but calls to \*real\* code
- dagger run go run ci/main.go build
test:
script:
# no funky untestable shellscripts here, but calls to \*real\* code
- dagger run go run ci/main.go test
```
> but is very annoying if you use Gitlab CI's built in parsing of junit/coverage/... files, since you now have extra layers of context to dig trough when tests fail etc
You can also still keep using these. The only thing you need to be aware is to export the required test / coverage output files from your Dagger pipelines so Gitlab can use them to do what it needs.
> but I've just written quick-and-dirty scripts to do that every time I've needed it.
This is what we're trying to improve. Those quick-and-dirty generally start very simple but they become brittle and very difficult to test by other engineers. Yes, you could use docker or any container-like thing to enable portability, but you'll probably have to write more scripts to glue all that together.
Quoting one of our founders:
"Our mission is to help your teams to keep the CI configuration as light and "dumb" as possible, by moving as much logic as possible into portable scripts. This minimizes "push and pray", where any pipeline change requires committing, pushing, and waiting for the proprietary CI black box to give you a green or red light. Ideally those scripts use containers for maximum reproduceability. Our goal at Dagger is to help democratize this way of creating pipelines, and making it a standard, so that an actual software ecosystem can appear, where devops engineers can actually reuse each other's code, the same way application developers can."
I think this comes close for Dockerfiles: https://hadolint.github.io/hadolint/ Just have to write a pre-commit hook for it.
I stopped working on it because dealing with shell shenanigans in the GitLab CI runner environment is such a miserable experience that we're in the process of moving all our jobs to python scripts.
- repo: https://gitlab.com/engmark/shellcheck-gitlab-ci-scripts-hook
rev: 174fda8f384db229aca5b40380e713d0c25a1cb9 # frozen: v1
hooks:
- id: shellcheck-gitlab-ci-scripts
files: ^\.gitlab-ci\.yml$
[2] https://gitlab.com/engmark/root/-/blob/9f7d9b93c2297d0b170e5...
# shellcheck source=./deployment/deployment-example.env
. "${1}"
For example:
set u
ignored_users=()
for i in "${ignored_users[@]}"; do
echo "$i"
done
Shellcheck also doesn't catch the expansion of variables as key names in testing assoc arrays:
declare -A my_array
un='$anything'
[[ -v my_array["$un"] ]] && return 1
un='$(huh)'
[[ -v my_array["$un"] ]] && return 1
-bash: huh: command not found
$ declare -A my_array
$ my_array["key"]=1
$ [[ -v 'my_array["key"]' ]] && echo exists
$ [[ -v my_array["key"] ]] && echo exists
$ [[ -v $my_array["key"] ]] && echo exists
$ [[ -v "$my_array["key"]" ]] && echo exists
$ bash --version
GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)
Disable default checks or enable optional ones using directives: https://www.shellcheck.net/wiki/Directive
The error checks can be pretty arcane: https://github.com/koalaman/shellcheck/wiki/Checks
I appreciate that the text for each check is brief and usually includes a suggestion. I end up disabling 26xx's a lot (for unquoted variables to be interpreted as multiple values).
Python is probably the best alternative to bash, but Swift is getting surprisingly good.
With shwift[1] you get NIO/async APIs, operator overloading for shell-like locutions, and trivial access to existing executables:
/// Piping between two executables
try await echo("Foo", "Bar") | sed("s/Bar/Baz/")
/// Piping to a builtin
try await echo("Foo", "Bar")
| map { $0.replacingOccurrences(of: "Bar", with: "Baz") }
For cross-platform use, be sure only use libraries on all platforms (i.e., not Foundation). It's a pain, but at least the error shows up typically at compile-time instead of run-time.
[1] - [shwift](https://github.com/GeorgeLyon/Shwift)
[2] - [clutch - any Swift scripts in a common nest](https://github.com/swift-nest/clutch)
I've had great experiences with this tool, but, for some reason, this issue always makes me question taking too great a dependency on it.
They're sponsoring quite a few devs. Caddy, curl, and SeaweedFS notably.
alias shellcheck='shellcheck -o all -e SC2292 -e SC2250'* https://www.shellcheck.net/wiki/SC2292
SC2250: Prefer putting braces around variable references (${my_var}) even when not strictly required.
- It obscures actual uses of modifiers, particularly ${foo-} when set -u is in effect,
- It's obvious when a name runs into subsequent text, even if one has somehow avoided syntax highlighting,
- And expansions followed by identifier chars don't actually occur in practice. Cases where the quotes cannot be moved to surround the variable are often interpolation of an argument to echo, whose behaviour is such a mess not even portable between bash and dash that shellcheck ought to be demanding printf at all times instead!
Useful: $UPPER_CASE for exported variables ("super globals"), $lower_case for anything else. Can also use $lower_case for function locals and $UPPER_CASE for exported and script global variables (stylistic preference; both are reasonable).
Not useful or reasonable: $ALWAYS_UPPER_CASE_NO_MATTER_WHAT.
I suppose people started doing it because they saw $EXPORTED_VARIABLE and thought "oh, I need to always upper case it", not realizing what that meant. And then after that more copy-copy of this "style".
bash is infectious in the bad sense :D
Realistically, Perl or Ruby would fill this role fine, but I hate adding another language to a project just for that purpose.
This makes shellcheck treat the file as bash script, so there are some false positives for zsh-specific syntax, but 99% of the rules work pretty much the same.
Special variables like $argv and $status.
Most array things (e.g. ${(o)arr} for an ordered array)
(( .. )) conditionals
"short loops" without "do".
repeat loops.
You could rewrite these things, but what errors remain where shellcheck is helpful in zsh?
"what errors remain" is hard for me to say – I get a wide variety of helpful errors, regularly.
It replaces a lot of uses of cut and "awk '{print $2}" for starters.
Most people don't even know the language has expansion rules and write stuff that accidentally works after the fourth try. This lang wants to become obsolete.
* Installing or starting your general-purpose runtime, passing the right configuration options
* Project glue where introducing general purpose language makes things more complicated because of tooling or startup.
* Simple cloud-init scripts
* A command line tool is simply the best library for the job and you don't want to get a general purpose language just to immediately shell out.
Old tumble dryers, vacuum cleaners - sure.
A simple example: I like to have some electricity switch/breakers automation, the best I've found are from Shelly Pro series, witch have a not really useful webserver built-in and not really useful APIs the rest are even worse having no wired versions at all. Why the hell not offer manual breaker + two wires for modbus so I do not need to fit ethernet wires and power in the same place?
Why just finding classic ModBUS-tcp/MQTT wired devices is so hard?
Things meant to be integrated does not need shiny UIs, need effective ways to integrate them, simple and reliable coms. My VMC witch is not a dirty cheap device have mobus support, unfortunately even the vendor do not know a full list of all registry and many of them does works "sometimes" like "write a 1 to switch from heating to passive ventilation", "sometimes works", so to integrate it I need to check if the command was "accepted" after 30" then re-check it after 40 because sometimes it flip back for unknown reasons... And the list is long...
MQTT is not a good solution. It's a very ad-hoc protocol that really only works if you are a programmer setting up a completely custom system. Obviously manufacturers are not going to cater to that miniscule market.
Or, pre compile it across all target platforms and have an install process.
BTW in a broad topic: a classic system with a user-programming language as the topmost human computer automation/interface is obviously better, but we have had such systems in the past and commercial reasons have push them to oblivion so...
If that’s what you need to do, C/go won’t only be much longer code, but also much harder and error prone. Command line tools are complex to deal with and there’s no magic bullet language for that.
You'd write a Python3 script I assume? With subprocess?
What makes this a plaything?
Did you respond to the right comment?
> a script that counts the number of lines in files versioned by git
I'm saying that is not a realistic production program that most of us would need.
If you want it as a personal utility to use in your own shell, absolutely you can use bash. I'm responding to the idea that such a trivial script would have long term use in production.
However, let's not produce any software at all, in case someone introduces a bug in it later.
Only winning move
1. seems to frequently get used for large products where people should really do strong static typing
2. Python 2 to 3 fiasco
3. Packaging and venvs and dependencies are a mess.
3a. versioning of dependencies. culturally handled in a slapdash kind of manner.
3b. Managing multiple versions of Python on my system, and multiple venvs per version.
5. Insanely, ridiculously slow compared to something like C. Slow even by the standards of scripting languages.
6. Python culture doesn't seem to acknowledge that regular users that don't care at all what language you used and want an executable, not a hot-glue build-your-own-environment hobby kit.
7. Pervasive culture of weak backward compat in libraries.
8. The pip devs are opinionated, and frankly user-hostile in my opinion, breaking backward UX convention, deprecating, general high-handed ivory-tower dev behavior.
9. I'd complain about the GIL, and a lot of people do, but I actually personally don't care. Maybe I covered it with the slow runtime and care indirectly.
Every time I encounter something written in Python I just mentally groan and roll my eyes. I genuinely don't like Python. Some of what I enumerated obviously bleeds into culture and ecosystem, but in practical terms for practical use, that's not so cleanly separable from the language in a lot of cases. Perl was always great about backwards compat and documentation, Ruby had a strong web dev bent because of rails, blahblah.
And as not many people use it as an sh alternative, not many complaints about it.
Something as simple as `./foo.sh file.txt | grep bar | tail -n1 | awk '{print($1)}'` would look ugly and convoluted in python.
About a decade ago I worked with someone who'd say this when a PHP project hit some ridiculous bug, and I'd suggest we maybe consider something else for future projects. Like, sure people complain that mixing tabs with spaces causes an invisible syntax error in Python; but at least it (a) has a dictionary datastructure, and (b) doesn't unavoidably convert some string keys into numbers, such that CSS colours like "112e33" become 112×10³³. (I thankfully don't use PHP anymore; inb4 "ackshually PHP7 is great"...)
What happens when you hit another bug in the new ecosystem?
If only it were "a bug". Rather, there was a steady trickle of post-mortems for production issues which couldn't be explained-away as a lack of testing; which secure coding practices would not have prevented; which took multiple experienced developers a while to figure out, and whose behaviour surprised all of them; and which were unavoidably baked into the design decisions of the underlying technology. This was also in a team that purposefully focused on building small, task-specific, loosely-coupled systems.
> is switching ecosystems and spending years building up your expertise?
Meh, it's easy enough to jump between PHP, Perl, Python, Ruby, Javascript, etc. since their differences are pretty minimal. Indeed, their language roadmaps seem to be "copy each other's homework" (generators, async, etc.; JS even added a `class` keyword FFS!). Even for someone who's only experienced with one particular approach taken by some specific framework, there's probably a work-alike clone to be found in all of those languages. Heck, the current PHP advocates keep going on about how different PHP7/8 are from PHP4; in which case, it may be easier to jump ship to another interpreted, untyped, imperative, procedural/OO scripting language (which may also be more battle-hardened and stable than that new version of PHP!) It's certainly not like switching from, say, Mercury to Zig!
Anecdatally: My first job was PHP programming, which I landed despite having never used it before; I was hired since I knew Python, I did fine. I got another PHP job a year later, whose interview included a written test. A few years later I learned that (a) its purpose was to check if candidates were lying about knowing PHP, by asking questions that exposed PHP's unique quirks and pitfalls; and (b) I'd apparently scored higher than anyone they'd interviewed before or since, despite only having used it for a year. In other words: some programming languages are so similar that switching between them may be easier than learning a new framework.
> What happens when you hit another bug in the new ecosystem?
My whole point is that this question is not answerable, precisely because "bug" is such a broad category. If the bug is a minor annoyance, we'd work around or avoid it; if it was more serious, but small, we'd maybe upstream a patch; if it was more difficult we'd maybe sponsor a core maintainer to work on it; etc. Presumably we wouldn't be hitting fundamental design problems, since our switching process would not land upon a notoriously flawed technology, cobbled-together by someone who "hates programming" and is "really, really bad at writing parsers"[1] (yet nevertheless proceeds to bake much of their language's semantics into the parser![2])
[1] https://en.wikiquote.org/wiki/Rasmus_Lerdorf
[2] For example, all the syntax errors and ambiguities I listed at https://stackoverflow.com/a/24463174/884682
If you don’t have to, you should never use it.
Just like any other tool, commit the time to learn it instead of just complaining it's hard.
Developers tend to think they can write amazing things with minimal effort and then curse the tool/lang when things turn out different.
The world runs on C and bash scripts... and it's just fine.
I did. Turns out I don't like it. What I dislike the most is that it brings out the clever type who likes to show off their tricks. That includes me. I've written atrocious Bourne scripts, and I've watched other people defend line noise in code reviews.
> instead of just complaining it's hard
So what's good about it that I should tolerate the weird expansions and arcane scoping?
> The world runs on C and bash scripts...
Less and less, thankfully.
> and it's just fine.
It may have been fine in 1980.
My toy OS has the most spartan shell imaginable (no pipes, no redirection, no variables, no flow control, just run a list of commands). It's enough to mount filesystems, configure network interfaces, spawn some services, and do the reverse for shutdown.
Complex tasks are delegated to Lua scripts or actual binaries instead.
I find that I'm okay with shell scripts which invoke repetitive individual commands, run complex piped commands, or run a sequence of commands with little-to-no control flow.
However, then I find that people use shell scripts for more advanced programming "because it's there" and "it's ubiquitous". What follows is a lot of stringly-typed programming using more and more bash-specific features and advanced control flow. External commands and builtins gather exotic flags to handle whitespace (or preferably null bytes). Then people want complex data types so they resort to things like `jq` and JSON stored in variables. But hey, at least they didn't use Python or Ruby!
Some people argue that by using a programming language like Python or Ruby or Go instead of "just a shell script" that I then bring about dependency management hell because I might use non-stdlib libraries and I require a minimum version of the language. But the people writing shell scripts rely a lot on their ambient system environment that they take for granted. They will use bash instead of sh and assume a recent version of it (good luck macOS users!). They will assume that the coreutils are BSD or GNU and they will get it wrong. They will want advanced data structures and assume you have some random utility for manipulating JSON, CSV, or TSV installed. Now you have a dependency management problem just like you would in other programming languages.
So then people say in a slack channel "hey, please install this util on your system", or they share Brewfiles and dockerfiles and container images, or they write yet another script to check your system for the right dependencies. You end up with a library of scripts and functions which require certain env vars to be set so they can all find and `source` each other. People writing and committing clever and complicated shell scripts usually end up with the same problems they would have with a different programming language that has package managers. But it's weeks or months down the road and it's easy not to think about when committing a 100 line bash script to the repo or adding new functionality to a script or depending on "just one more" installed utility.
It devolves to the same problem of "I'm writing Python/Ruby/Haskell/C/C++/... and I'm relying on the operating system's package manager to provide the libraries I depend on", but without having a powerful general purpose programming language with an import system and featureful standard library. Some teams do that and shell scripts which rely heavily on the system environment might be a great fit for them.
The best use case I can think of for shell scripts are FreeBSD init scripts, but then I consider how tightly coupled those are to the operating system. So basically... part of the system.
I don't think I have the discipline for all of that, so my flowchart for writing shell scripts is: is it a one-liner? If yes, do it. If not, use the same programming language as the rest of the project(s) in your team.