Hacker News Reader

Comment by giancarlostoro | Hacker News Reader

Any recommended reading for Bash? I'm somewhat new to it and it's interesting ways of getting things done. I've used it minimally in the past, but have found myself writing a 100> LOC script, which I can't help but feel I'm likely over-complicating certain bits and pieces.

View on HN

I'm a fan of reading code.

I would generally describe a GNU/Linux distro as being a "giant pile of shell scripts". That's a little less true with init scripts generally now being systemd units. But that's where I'd start: Look at the code that distros write, that isn't part of some other upstream software.

- Arch Linux's `makepkg` https://git.archlinux.org/pacman.git

- Arch Linux's `mkinitcpio` https://git.archlinux.org/mkinitcpio.git/

- Arch Linux's `netctl` https://git.archlinux.org/netctl.git/

- Downstream from Arch, Parabola's "libretools" dev tools package https://git.parabola.nu/packages/libretools.git/ (disclaimer: I'm the maintainer of libretools)

Gentoo also has a lot of good shell scripting to look a, but it's mostly either POSIX shell, or targets older Bash (their guidelines http://devmanual.gentoo.org/tools-reference/bash/index.html say to avoid Bash 3 features). I tend to believe that changes made to the Bash language over the years are enhancements, and that they let you write cleaner, more robust code.

There used to be this great resource called Linux Documentation Project. It's not as active as it used to be but it produced some really book-quality documents, including the Advanced Bash Scripting Guide at https://www.tldp.org/LDP/abs/html/ .

Read it! It's great. But know that a lot of bash scripting isn't really in bash, it's really required to be proficient with grep, sed, cut, dc and a few other text processing utilities. Learn those! Then are are a few other tricks to be mindful of: be mindful of spaces (wrap your variable substitution in double quotes, mostly), be mindful of sub-shells (piping to something that sets variables can be problematic), and a few other things that can really only be learned by reading good code.

But it's also good to know when you shouldn't venture further. My rule of thumb is when I'm using arrays or hash maps, then it's a good idea to move to another language. That's probably Python nowaways. A lot of people use tons of awk or perl snippets inside their bash scripts, that can also be a sign that it's time to move the whole script over.

Advanced Bash Scripting Guide on TLDP is a verbose and cumbersome-to-read writeup. Better read the bash man page and then http://mywiki.wooledge.org/BashGuide and http://wiki.bash-hackers.org/start

It can't be done. If you want to write reliable code, and actually notice all of the possible error conditions instead of silently ignoring them, your code needs to get more verbose and complicated than it would be to just use a more capable tool like Python or Node, and it still won't be as reliable.

If you have more logic than a couple of string comparisons, Bash is not the right tool for the job.

As they say, bash is just good enough not to get replaced.

I recommend Greg's Bash Wiki ... https://mywiki.wooledge.org/BashGuide. See general notes, then at the bottom of the page are many links to additional considerations.

Like others say, "bash" is a hard tool to get right (and I'm not saying I do it right either, necessarily, but Greg's Wiki was real helpful!). I'm building a hybrid bash/python3 environment now (something I'll hopefully open-source at some point), and bash is just the "glue" to get things set up so most aspects of development can funnel through to python3 + other tools.

But ... things that make bash real useful:

    * it's available everywhere (even in Windows with Ubuntu-18.04/WSL subsystem)
    * it can bootstrap everything else you need
    * it can wrap, in bash functions, aliases, and "variables" (parameters), the
      real functionality you want to expose ... the
      guts can be written in python3 or other tools

Without a good bash bootstrap script you end up writing 10 pages of arcane directions for multiple platforms telling people to download 10 packages and 4 pieces of software per platform, and nobody will have consistent reproducible environments.

EDIT: I think there's a revised version of Greg's Bash Wiki in the works.

Or you just bundle your application and its dependencies into a single folder for each OS and distribute it.

It is availbale, almost, everywhere but be careful with the version, different Linux diatros are at different versions, the last time I used OSX it was stuck on a very old version, and I expect the different BSD OSes to run fairly new versions.

BSDs don't ship Bash as part of the base system - you have to install it from packages or ports. And that one is the most recent that the maintainer bothered to package. E.g. FreeBSD is on 4.4.23 right now, which actually appears to be newer than e.g. Debian unstable.

>Python or Node

Python, sure, but replacing Bash with Node just seems like replacing a language crippled by its need to be backward compatible with (Bourne) sh with a language crippled by its need to be backward compatible with what Brandon Eich came up with in two weeks in 1995.

It wouldn't be my first choice either, but modern JavaScript isn't backwards-compatible, it is at least possible to write reliable and robust code if you take some care and know what options to enable, and it's mostly consistent across platforms. Last month, I spent a week trying to handle all of the possible error conditions in an overgrown bash script. It was immeasurably worse.

Python is the more direct substitute, though: it's built-in to almost every platform, and the Python 3 stagnation has even given us a consistent version: 2.7.

Sure it can. Treat it as a functional and immutable language, and you can go pretty far with it, pretty safely.

You're forgetting that bash really just calls other code. So you can combine two language's stdin/out functionality (eg, Python or Node) to pair code together. Sure, it won't be fast, but it can do quick wonders as an ad hoc data pipeline.

I just use set -e at the top of the script

That's a good idea, but it's not a complete solution. See http://mywiki.wooledge.org/BashFAQ/105

Nice. But I use C++ so I'm used to shooting off my own foot.

By far my best advice is to write auto-complete scripts if possible: https://iridakos.com/tutorials/2018/03/01/bash-programmable-... . Many of mine will do things like look up database values, so "command<tab><tab><tab>" is the workflow 99% of the time.

Keep them short. There are always exceptions but the "do one thing" mantra is handy, they can always be wrapped into a more complex workflow with a bigger script. None of my frequently used ones are over 100 LoC.

Write them for you and you alone when possible, start off simple and iterate. Fight that developer urge to solve a generalized problem for everyone.

Embrace the environment and global environment variables. We're trained to avoid global variables like the plague but they are really useful in scripts. My auto complete scripts that I mentioned above, they know which database to connect to based off the environment variable and there are separate commands to switch environment.

Make sure you aren't using it where things like make would be more appropriate.

I recommend Greg's Wiki. It's the only resource I use. It covers common pitfalls and anti-patterns.

https://mywiki.wooledge.org/

I've recently picked up a subscription to Destroy All Software and he has a few really good demos of how to architect simple and powerful shell scripts using bash. I'd really recommend it for anyone who doesn't need help with the semantics and instead needs help with the organization of a script.

I've got a subscription too, watching him go in some of the videos as he casually does Bash blew my mind, which is another reason I ask this question. I'm trying to at least watch all his videos, I intend to work through them after watching them all at least once, and then I intend to rewrite some of the solutions (the Ruby ones) in Python and Go just to be sure I understand the concepts more, and I'm not just copying and pasting.

* look into parameter substitution https://www.tldp.org/LDP/abs/html/parameter-substitution.htm...

* use traps http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_12_02.htm...

* read about safe ways to do things in bash https://github.com/anordal/shellharden/blob/master/how_to_do...

* this is pretty helpful writeup of general CLI usage, yet not bash specific https://github.com/jlevy/the-art-of-command-line (related HN discussion https://news.ycombinator.com/item?id=9720813)

From the google Shell Style Guide: "If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. "

https://google.github.io/styleguide/shell.xml

LOC alone doen't mean much. I have a backup script in fish of 400+ lines of code because it tries to dump data out of different sources like MySQL, PGSQL, InfluxDB, /etc files and others. Just use what feels comfortable achieving the task.

That guide is really useful for making scripts more readable.

I recommend reading and memorizing all 30-ish of the readline shortcuts

https://tiswww.case.edu/php/chet/readline/readline.html#SEC1...

Readline is the library bash uses for editing the input line, and it has some nice movement keys. For example Alt-b moves the cursor back a word, Ctrl-u deletes to the beginning of the line, Ctrl-w removes one word behind the cursor.

They work in a bunch of other programs, like the Python interpreter's interactive mode for example.

Excellent advice! Thanks.

> have found myself writing a 100> LOC script

Often, this is a good sign that you might want to switch to another scripting language.

That's true, so many times I have used shellcheck and started to improve a bash script, only to realize my time was better spent rewriting the script in Ruby.

Take a look at this, as well:

https://amoffat.github.io/sh/