Common Bash Pitfalls | Hacker News Reader

Common Bash Pitfalls

mywiki.wooledge.org

60 pointsdunk01025 comments

It's the problem of nested DSLs (sed, tr, globing, bash's own control commands etc): you need to quote (escape) each DSL's meta-characters like crazy.

It's a scary, horrific nightmare. Sure, you can learn all the intricacies, but what a waste of your cognition! There must be a way to do it with clean abstractions (assuming the unrealistic liberty to redesign all unix commands). But... I have a suspicion that the horror is intrinsic to the problem, and bash strikes a legitimate trade-off between brevity and safety. It would take some work to be sure about this.

Based on no evidence whatsoever, my gut tells me that trying to set up a tidy system of abstractions to clean up the problem would produce something closer to a programming language than a shell language. On the other hand, it's possible that exposure to *nix has simply impoverished my imagination when it comes to things like this.

The quoting rules in the Plan 9 shell rc are much saner. Bourne and C shell quoting rules are a recipe for trouble. Sloppiness (e.g. leaving out quotes, using the wrong kind of quotes) works about 99% of the time which leads you to think your code is correct.

The big difference is that rc does not expand the result of variable expansion. I've been planning to write a rc derivative that had job control for a long time now. So far, I've never had enough spare time.

Is Bash scripting still relevant today, other than working with and understanding legacy code? Why wouldn't I use something like Python or even Perl instead?

Shell scripting is as relevant as it ever was.

Python or perl scripts are fine if you are doing more complex and "algorithmic" stuff, but for automating system administration tasks, I often find that shell scripts are simpler and more concise.

But shell scripting isn't just "scripting" (as in ".sh" files with commands). A proficient unix user/sysadmin will often build one-shot commands using features other people thing are only useful in batch scripts. Actually this is something I find baffles both the Windows-types and the people that solve every problem with "perl -e".

You're right about people needing to understand bash and Unix commands - I've seen two 20+ line Perl scripts that run the equivalent of 'date -I' and 'nc' respectively.

That said, I believe bash is somewhat limited for system administration - which I know is very odd amongst system administrators - but please let me explain why.

Using the shell on a Linux based OS in 2009, you miss out on:

* filesystem events (inotify)

* hardware events (dbus)

* config for programs with tree structured configuration (lxml)

* config for programs with sqlite based configuration

* RPM / yum beyond what their command line apps expose (and perhaps dpkg/apt too, but I need to investigate this further).

...as the commands / shell function libraries to handle these are often immature compared to the equivalent APIs. As a result, you get sysadmins causing unnecessary load by polling a file repeatedly at intervals (eg, via a cron job) rather than letting the kernel tell them when their file has changed.

I'm hopefully giving a talk about using richer languages for system administration at PyCon 2010 - I'd love to hear any thoughts or opinions on these matters fron HN readers.

Apart from the reasons CrLf was pointing out, here is another one: The shell (and a set of utilities) is standardized - it's part of the POSIX standard - this is important for some organizations. If you program in the subset of bash that conforms to the standard (enforcable with bash --posix) you are programming against an interface that will likely be more stable over the coming years than Python or Perl (think Python 2->3, or the long awaited Perl 6).

Standard Shell Command Language: http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_...

As always, it's a tradeoff, i.e. the best solution depends on the problem and requirements. Many problems shouldn't be tackled with the shell, for sure...

I do it every day using Cygwin. Certain tasks are so much quicker when you can for/while/do/cut/grep/sort/uniq/wc/>< and so on.

In my experience, the main advantage of bash is conciseness for certain tasks. I don't have much experience with Python, but I have found the shell (and associated command) less verbose than Perl in certain cases.

Obvious things like "find" and "grep", but also things like doing things with the output of a command. In Perl, you need to open the command as a file handle, which takes a couple lines of code. In bash, you just use "|".

I wouldn't write a multi-thousand program in bash, though. Not again, anyway...

Good question. I can think of two tasks for which I routinely use the shell, and wouldn't really prefer one of the newer scripting languages:

1. Customizing my interactive shell, although I personally favor zsh over bash. I write shell functions and automate PATH setting and discovery where possible, so knowing how to script the startup file helps.

2. Startup scripts for software I distribute. Perl or Python would work, but then I would have to be careful about which version of the language to use. With /bin/sh, as long as I use purely POSIX constructs (which means no arrays, unfortunately), scripts work anywhere.

Thanks for this post. I'm new to doing bash scripting and this is very helpful.

I've got the O'Reilly bash book, it's well worth getting.

the only university text book i've held on to.

The only one I didn't know about is the problem with:

     echo "hello world!"

I'm surprised history expansion could reach inside the quotes.

You need to use single quotes. !, $, `, \ are expanded inside of double quotes.

I know about variable and command substitution and escaping, it makes sense that those expand inside " but not ' - but history expansion, something seemingly inherent to the interactive REPL, seems out of place here.

Special characters are substituted in double quotes. They're treated as literals in single quotes.

Seems pretty consistent to me.

Aliases are handled differently though, before e.g. file glob expansion. I would have expected history to be handled similarly.

I understand your comment. Intuitively I agree with you but in the context of one of the original devs writing bash it would make no sense to tie the history command to the interactive REPL. Treating everything the same would lead to less bugs in "their" system while leaving a caveat for all the userland script writers.

There is no difference between a script and the REPL.

Besides, you'd expect variables to be expanded inside normal strings -- why special-case the other expansions?

I'd expect them to be handled in the same "phase" as alias expansions. See here for one example of hackery with aliases, that take advantage of their different expansion phase:

http://www.chiark.greenend.org.uk/~sgtatham/aliases.html

Most of those aren't specific to bash.

Correct -- many of these are also problems in Bourne or Korn shell.

I guess by virtue of being the default shell on most Linux distros, "bash" is now seen as a synonym to "Unix shell programming".

I see it more as: people who have problems while working with Bash will go to the Bash wiki, even if the mistake they're making could be made in other shells.

That's probably the case. Also, all of them being supersets of the Bourne shell doesn't help either.

Reply on news.ycombinator.com