Bash code generator for command-line arguments | Hacker News Reader

Bash code generator for command-line arguments

github.com

78 pointsvitro51 comments

I start all my bash scripts from the following template, then modify it as needed. It is not fancy, but I think it supports the lowest common denominator that is compatible with other flag processing systems in all other languages that I am aware of. (In particular, it does not support '--option=opt' but uses '--option opt', which is cross-compatible). I like that it is small, easy to maintain, and avoids external dependencies in my shell script:

Edit: Send error messages to stderr

Edit2: People seem to like it. I created a GitHub gist for it here: https://gist.github.com/bxparks/e67a3d6fc6b5d62b51304b3d9de2...

    #!/bin/bash
    set -eu

    function usage() {
        echo "Usage: parseflags.sh [--help|-h] [--binary] [--option opt] [--] files..." 1>&2
        exit 1
    }

    binary=0
    option=''
    while [[ $# -gt 0 ]]; do
        case $1 in
            --binary) binary=1 ;;
            --option) shift; option="$1" ;;
            --help|-h) usage ;;
            --) shift; break ;;
            -*) echo "Unknown flag '$1'" 1>&2; usage ;;
            *) break ;;
        esac
        shift
    done

    echo "binary=$binary"
    echo "option=$option"
    echo "files: $@"

In my experience this is the simplest way. The only mod I’d suggest is to put positional arguments into an array so you can put flags anywhere but this will work fine for 99% of scripts.

The problem I have with bash arrays is that I use them so infrequently that I cannot remember its syntax. I have to look up the manual each time, and it's difficult to find things in the bash man pages.

This is nice! I suggest writing messages to standard error instead of standard output.

You mean like this?

  echo "message" 1>&2

Yes, that's probably a good idea, though I cannot recall this causing a problem for any script that I've written so far.

Wait, --help should print to stdout so that

    fancycommand --help | less

works as expected. Otherwise, the help text will dump to the screen and less will erase it when it decides to redraw its empty buffer! Actual error messages should go to stderr so that I can redirect them to an error log or so I can suppress them if need be, or I could also be doing > /dev/stdout to silence normal output but I don't want errors to be discarded.

I agree. When a program is given the --help option, the usage information is the actual output.

It allows users to easily separate messages intended for the user from output which can be consumed by other scripts.

  $ tens.sh --verbose 5
  10
  100
  1000
  10000
  100000
  Calculated 5 results

  $ tens.sh --verbose 5 > results
  Calculated 5 results

This is handy, thanks! Definitely one of those things I wind up badly/slowly reinventing the boilerplate for every time.

I recommend making 'usage' return 0, rather than 1; since asking for help is valid.

(Since you use it both for -h and for unknown options, perhaps make it not exit at all, and make the caller exit with an appropriate code?)

Additionally, you should output the actual executable name (as passed in on $0) rather than hardcoding something. So maybe at the top say 'executable=$0; shift'; replace parseflags.sh with $executable; and replace $1 with $0 in your option parse loop.

I think technically you are correct about --help being valid. But in practice, I use --help only in an interactive context where the exit code does not matter so I've never been bitten by the subtle difference.

I tried using $0 years ago, but I didn't like seeing the full path name (or the "./parseflags.sh" string), or maybe it was because I often use shell alias, so the $0 becomes the alias name, or something like that. So now, I just replace the string "parseflags.sh" with the name of the script for each script. I think I tried using $(basename $0) at one point, but didn't like that either, though I don't remember why.

I hope that the template is simple enough that people can customize it as they wish.

Edit: Fix typo

just fyi, `shift` doesn't affect $0, it slides the window of $1..$N back, so you don't want to shift after grabbing $0 and you don't want to change the option loop if you do this.

And you can just use $0 everywhere so there's technically no reason to pull it out into a $executable variable, but tbh that gets a bit confusing in functions.

> getopt is discouraged, getopts doesn't support long options

Hold up. The only version of getopt with long option support is from util-linux, which fixes all the whitespace issues people were discouraging getopt for. So what's the stated goal of this project again?

Heck, you don't even need to write code to detect usage of legacy getopt. It will panic every time you use the new stuff once it sees the option to turn on escaping.

Advertise all your cool features like argument types and help generation, but not this. I don't see how writing m4 is easier than just a while-case loop, but I may try it for the extra features.

Getopt isn't a builtin and the BSD/OSX one is different.

I'm probably going to get downvoted for this, but:

writing bash scripts in general just isn't a fun experience.

years ago, i started just writing ruby scripts for execution in the terminal instead of bash, and haven't had any problems with it so far.

A few years ago I would have disagreed, but today I quite enjoy it. What changes my opinion was mostly having a decent IDE configured to deal with the annoying bash syntax errors, and a quicker way to prototype.

Though in generqal I always thought Bash scripting to be fun, due to the wide array of programs available that you can directly invoke in your scripts script. The feeling of gluing programs together with pipes into a larger abstraction is just amazing, and bash does that very elegantly.

That last part is why I routinely stick with bash for quick scripts. For example, I usually create boolean values to check for a proper config by using grep or grep -c.

Do you have any recommendations for an IDE to use for Bash scripting?

Shellcheck (https://www.shellcheck.net/) is essential, with or without IDE.

Pycharm has it built in or as a plugin now too.

Yes, Bash is nice as long as you use it as a shell only.

For a scripting language, better look elsewhere.

If we had to standardize on one language, perhaps Python would be a good choice.

Ruby is definitely a better tool for this but there are situations where it's not available but bash is. That portability is really the only advantage.

Now that Python 2 is deprecated, I think that settling on Python for scripts is the happy medium. It is available in all but the most stripped-down linux builds, and those are commonly seen today only in docker images or hardware. And adding it to e.g. Alpine is trivial.

I got bit by a Bash-ism late last night (-ne vs != causing a CI test to silently fail) and that was kind of the last straw for me.

I drive zsh but still use bash scripts a lot cause "portability" and "it's available everywhere" but that argument just doesn't hold up when I'm already in control of most of my stacks and have docker access on pretty much every host I'm in.

I'm forcing myself to use Xonsh more, which despite having all its own quirks, is so much less mental overhead going between Python development. Yeah maybe it's a few more steps (than zero for bash) to provision, but it's worth it.

Like @bxparks, I have a template for my bash scripts command-line parsing, but I like to stick to GNU-style options and I find that Bash's getopts command handles all these cases without trouble:

command -ffilename -f filename --file=filename -- -f is not an option

My template is at https://gist.github.com/webb/ff380b0eee96dafe1c20c2a136d85ef....

Using `-:` to build long-options on top of the short-option-only `getopts` builtin is a great hack.

Thanks!

Quite an impressive project all things considered; definitely something to keep in mind for the next project that inevitably starts out as a simple Bash script.

Out of curiosity: if you (the reader) get to a point where you need more complicated and esoteric functionality that Bash can provide, do you still keep hammering at it or turn to a more capable (and readable) language?

Just wondering if people are usually working in very constrained environments where other languages aren't able to be readily used.

I've tried that path with little success. Usually, arg parsing makes me try some other language (perl, python) but other things got complicated there, mostly running other processes, capturing their exit status, pipe-ing their output etc.

In the end, it was easier to hammer onto bash. Also, there are great linters for bash scripts, so things work out.

tcl is designed for this situation

Thx, I’ll give it a shot!

> but other things got complicated there, mostly running other processes, capturing their exit status, pipe-ing their output

Note that Python's subprocess module [0] has received many updates since v3.5, and now I find it relatively painless. Most of the times, subprocess.check_output() or subprocess.run() will do the trick.

[0]: https://docs.python.org/3/library/subprocess.html

Thx, I’ll check it out again!

I've been in a similar boat, yet despite as my experience with it grows, bash just wears on me. Everything feels like a hack. Sure, piping is easy, but if statements are atrocious.

Starting to get into Xonsh. Still a learning curve, cause it's neither bash nor python, but it's really refreshing.

As soon as as the cli becomes non-trivial I switch to python and use docopt for that (really cool module btw.). Pipelining processes is not as idiomatic in Python as it is in Bash though, but everything is better than getopts. I'd rather decapitate myself with a teaspoon than using getopts.

Docopt exists (now, think it was originally python) for many languages including bash [0] (or perhaps it's POSIX, haven't checked).

I like it too, not least because I can use ~the same thing in multiple languages and not have to remind myself how the arg parser for language X works each time.

[0] - https://github.com/docopt/docopts

Yeah, I’m kind of shocked to see so many masochists in this thread. Built-in subprocess is nice and all but I’ve had a lot of joy using https://amoffat.github.io/sh/ to do the work (and the maintainer is super responsive). I sometimes don’t have the flexibility to have external dependencies but I’d use it everywhere if Python just folded it into the default library.

It's a trade-off. Especially if the script contains a lot of pipes and redirection, it's difficult to reproduce with the same elegance in something like Python.

My biggest successes have been extracting these parts into very small stand-alone shell scripts which the Python application then invokes, which gives you the best of both worlds. The best example off the top of my head is `mysqldump | sed ... | gzip` with decent error handling (i.e. set -o pipefail in bash).

My environment isn't all that constrained, but I do have to deal with old Python versions (though at least I don't have to support 2.6 any more...), and a surprising amount of the features that would make replacing bash easier (subprocess.call especially) are only available fairly recently.

I’m curious about your experience that pipes and redirects are harder with python. Do you mean slightly more verbose or just complicated and difficult?

I’m curious if you’ve tried the sh Python module [1]. In some ways it’s much prettier to read although the initial writing of it can be awkward to get used to at first.

[1] https://amoffat.github.io/sh/

I mean significantly more verbose and more complicated. Fee free to try it yourself - replicate `bash -eo pipefail -c 'mysqldump example_db | gzip > dump.sql.gz'` in Python with streaming (databases can be several GB, so you can't read it all in memory).

I have not tried any of these libraries, though they look nice. The only times I've had to write scripts that make heavy use of subprocesses and pipes, I want them to work with only the standard library so I can just rsync them and they just work™.

You have to dig through the documentation a bit on the website but all the info is there[1]

  from sh import mysqldump
  from sh import gzip

  gzip(mysqldump("example_db", _piped=True), _out="dump.sql.gz")

or if you don't want the magic import thing:

  import sh
  sh.gzip(sh.mysqldump("example_db", _piped=True), _out="dump.sql.gz")

It's definitely foreign to shell scripting languages since the piping syntax is a bit different & you have to remember to do `_piped=True` since it's not parallel by default. But the default behavior is pipefail & exit on the command failing IIRC so it's more of a choose your poison thing (do you want a subtle perf issue or a subtle bug in your script not handling error states correctly). And I find it easier to read. + if you want to customize anything about gzip or not rely on needing the binary in the path, then you can just switch it to Python-native gzip pretty easily.

That's my favorite feature. Conciseness if I'm just translating a script with progressive complexity options to migrate things that need more complexity or different requirements within the same script without having to rewrite it from scratch.

[1] https://amoffat.github.io/sh/

> My biggest successes have been extracting these parts into very small stand-alone shell scripts which the Python application then invokes ...

That's a clever way to go.

Sometimes the other way around works too: taking the most fiddly manipulations (say, date or time intervals) and re-homing them to a small Python script, but leaving the bash intact.

There is also plumbum for more ergonomic pipelines in python:

https://pypi.org/project/plumbum/

Speaking as someone who frequently injects the most ridiculous of things into a makefile in order to bootstrap build tools and such things... and as someone who had learned my way through enough of the Chrome OS build tool chain to customise a fork of it for building my own variant of CoreOS back in the early days before Kubernetes won the container platform wars...

It depends. Sometimes you use these tools because they are already responsible for so much stuff it makes sense to just extend it no matter how crazy it is (Chrome Os build tool was built as hundreds of lines of shell script which built on top of portage which is even more hundred of lines of shell script) ... sometimes it’s because that’s the tool you know will be available (the dozens of bootstrap scripts I’ve beaten crushed and smashed into makefiles complete with OS detection and other stuff) ... and sometimes you don’t have any read you just do it because it’s what your comfortable thinking in. I’ve definitely written bash scripts that would have been better in Python but at the time my mind was building the script up as a sequence of Unix tool operations and pipelines not as a Python program.

There's a bit of code duplication[0] that often emerges as a result of parsing command-line arguments in shell scripts. The duplication can be eliminated by defining[1] the set of arguments to pass into a reusable template script[2].

[0]: https://dave.autonoma.ca/blog/2019/06/16/typesetting-markdow...

[1]: https://github.com/DaveJarvis/keenwrite/blob/master/installe...

[2]: https://github.com/DaveJarvis/keenwrite/blob/master/build-te...

This is incredibly full featured.

I wrote a much less full featured and less ergonomic library that targets sh rather than bash. It was 70 lines of shell including whitespace.

Here's an example of its usage[1]. Each option calls a function, optionally with an argument. There is also sugar for options that merely toggle a value.

Non flag parameters call a function "positional parameter" to avoid a dependence on arrays.

It does not generate usage or man pages.

1: https://github.com/jasom/the-copper-searcher/blob/master/opt...

Argbash is great. I found the output unnecessarily verbose for the simple functionality that I wanted, so the way I've used it has been to put the following in a script/function and then eval the output: https://gist.github.com/buu700/72d0461d318bfe8c11e36d2316882...

That would then be used like:

    eval "$(parseArgs \
        --opt foo \
        --opt-bool bar \
        --pos balls \
    )"`

Yeah this problem with shell is real ... it ironically is not good at writing command line interfaces!

If anyone is interested in Oil, I would appreciate help either:

(1) Running this library with OSH. Try it and tell me what breaks! OSH runs many unmodified bash scripts.

(2) Help with a new builtin that will address the deficiencies of getopt/getopts. You could help design it!

https://github.com/oilshell/oil/issues/469

Awesome! I think it would be even better if it could generate from doctopt style definitions :p

> getopt is discouraged

By who? Why?

This is the version that I’ve always used:

https://manpages.debian.org/buster/util-linux/getopt.1.en.ht...

Start at https://unix.stackexchange.com/q/62950/5132 .

Summary of that reference: If you’re on Linux, then you’re fine, and getopt(1) is good.

what a gem!

Reply on news.ycombinator.com