The importance of user interface, illustrated by the Go flag package
utcc.utoronto.ca
utcc.utoronto.ca
I wanted to create command line programs that look like traditional GNU/Unix utilities, so I wrote the https://github.com/ogier/pflag package. It's a drop-in replacement for the flag package, just change the import line. It also adds some basic features, such as the common pattern of a flag with a long-hand form usable with a double --dash and a shorthand character with a single -d.
The Go flag package's command lines adopt the observation made by the Google C++ command line flag parser (https://code.google.com/p/gflags): the long/short --/- two-names-for-every-flag dichotomy was useful for backwards compatibility when recreating original Unix commands, but for new programs, it doesn't pull its weight. At the cost of requiring that each option be specified with a separate command-line argument, you can drop the distinction between - and -- entirely. (This is similar to what X11 programs do, although I believe they do not admit --x as a synonym for -x.)
It's not clear to me what is "ugly" about not having to keep alternating between - and -- as you spell out options on the command line.
Short is more usable for users on the command line. Long is more maintainable in scripts. That's why it makes sense to have both.
> At the cost of requiring that each option be specified with a separate command-line argument, you can drop the distinction between - and -- entirely.
Again, that cost is high for users on the command line, one of the few places where every keystroke does matter. It's a lot more pleasant to be able to do:
$ rm -rf
$ rm -r -f
$ rm --recursive --force
Ultimately, you're interfacing with a greater primate. Even with all of our nice features, we still don't support USB or web standards. Instead, it's all lossy OCR and weird muscles. You should expect a certain amount of... squishiness... when interfacing with something like that.
My officemate is a long-time Git user but was surprised the other day to see me type "git pull -r". He didn't know that -r was short for --rebase, and in fact I didn't know that --rebase was long for -r. This kind of mutual incomprehensibility is a tiny variant of the observation that every C++ programmer uses only a small subset of C++, but the problem is that each programmer chooses a different subset. Not having two names for a flag can be a feature.
Re typing, I am also not convinced that the overhead of rm -r -f vs rm -rf is really that high. Of course, rm is a bad example: if you are implementing rm, you MUST use the Unix conventions and accept -rf. But for new commands not burned into decades-old muscle memory, it's not clear that it's needed.
I think there are multiple valid design decisions that could be made here, and we've tried to tilt the balance toward simplicity, but of course other balances could be struck, and maybe in other contexts it might make sense to strike a different one.
I tried to make sure it was possible that someone could write a package (say, flg) with the following API (and nothing else):
// Package flg implements a getopt-compatible command line flag parser.
package flg
// Parse parses the command-line flags defined in package flag.
// In contrast to flag.Parse, Parse imposes getopt semantics:
// - single letter flag names must be specified with a single dash: -x
// - longer names must be specified with a double dash: --long
// - the argument to a single-letter flag can follow it immediately:
// -xfoo means -x foo when -x takes an argument.
// - multiple short flags can be combined: -xyz means -x -y -z,
// when neither -x nor -y takes an argument.
// - name aliases can be introduced by calling Alias before Parse
func Parse()
// Alias records new as an alias for the flag named old.
// Typically old is a long name and new is a single-letter name or vice versa.
// For example, Alias("r", "recursive").
func Alias(new, old string)
It's about the principle of least surprise.
Regardless of whether double-dash opts came from UNIX or GNU (and obviously you are correct in that they came from BSD/GNU), they're the convention nowadays and it's less confusing for end users to just embrace the convention.
Go has bucked the (recent) tradition on a lot of different fronts, which I love it for, but the flag package -- the interface to the users of the software, not the authors -- is one place I don't really think it's quite as appropriate.
Certainly, Google is able to make these sort of changes in its internal world, and reach a high enough percentage of software that the new choice becomes normal, but Go will probably never have enough mindshare in the extraGoogle for its CLI parsing to not be "a little weird".
---
As an addendum, I happily use the builtin flag package for software that is consumed primarily by my team. For software I expect other teams to use, who don't necessarily care about the fact that it was written in Go, I prefer pflag.
One would think that if any OS would foster the idea of sameness it would be those belonging to the Unix family.
As for old vs new style command line options: that's now Unix in much the same way that Dave Gilmour was a proper part of Pink Floyd and not merely "the new guy".
The original OS we call Unix has not been relevant to the discussion for 20-25 years.
Thinking of Unix as a specific OS and a specific trademark in 2015 is just odd since the majority of people using various forms of Unix today were not even born when the distinction was even relevant.
And just as with other families, the peculiarities of a single member does not automatically apply to the entire family.
For many people, those sorts of commands are 80% of what they run on the command line. Having it so that the other 20% consisting of new programs works different is confusing and annoying.
Meaning that you can do things like ls -ltr to get ls in long format, time sorted and reversed.
Having to rethink my entire career of something simple like "how to pass flags to a binary" had better bring significant improvement, and flag does not. flag regresses a lot of convention about the operator interface to a program. This:
binary -vexd data output
binary -verbose -extra -one-file-system -data-dir=data -output-dir=output
Python argparse gets this nearly perfect, subcommands and all. flag is near the top of what I hate about Go as an SRE. Java too.
This is a somewhat educated guess. A long time ago at Google I used to work on the program that was used to launch most jobs on the clusters. To help mitigate the command line length problems, the launcher preferred generating command lines with -nofoo instead of --foo=false, and -foo instead of -foo=true. This saved a few characters per boolean flag. Of course it also caused no end of trouble for people who'd defined their own "three-state boolean" flags that could e.g. be true/false/autodetect. Not the finest hour of engineering, in retrospect.
Probably the only truly controversial bits of the Go standard answers are A: pulling in external packages isn't free and extremely frequently underestimated over the life of the product, they better be bringing some real value beyond merely preventing the developer's nose from crinkling a bit [1] and B: if you've only got three flags right now, don't go leaping to the conclusion you're in the process of writing a grep replacement; wait until you actually have the situation I described before dropping another package in (YAGNI).
[1]: Contrast this with something like Node or Ruby that seems to encourage pulling in dozens of libraries, many of which may literally be single-digit lines of payload code, and it is considered totally worth it for developer tastes to alter significant elements of the language/runtime/stdlib. And I really do mean contrast this with... I am currently expressing no opinion as to which philosophy is "better".
For a language like Go I also strongly question why keeping with the stdlib is necessary. There's value in a language like Python where you the environment is dynamic - Go produces statically linked binaries. Who cares what libraries you build them with? That's kind of the point.
I wouldn't deny that.
But I do agree with the Go philosophy (now I am expressing an opinion) that says you should also not underestimate the ongoing technical price of taking on a 3rd party library. Most developers consider it "free". In practice, it isn't. It is entirely possible for the value to exceed the cost, but developers rarely make a sensible decision... they just account the costs as zero and go grab the library, and often don't even notice what they're paying, and certainly don't notice what they're costing the team.
(Bear in mind we're talking "stdlib" vs. "grab a library" here, so NIH-syndrome is not currently on the table.)
And you're also really prejudicing the discussion, and indeed probably your own mental visualization, by speaking as if every Go program is a budding "command-line catastrophe" waiting to happen, when in fact BY FAR the most common case is that your program takes zero command line parameters, and the vast bulk of the remainder is five or fewer with no complicating factors like "modes" or something. Those who are programming the remainder are free to grab any of the 5 libraries I saw mentioned in the discussion yesterday (and probably more since I last looked), and nobody is going to complain.
"Who cares what libraries you build them with? That's kind of the point."
The maintainer. Go is generally focused on "serious" programs who will be maintained by teams of developers over periods of time that typically involve 100% turnover (though, hopefully, not all at once). If you don't have that problem, well, Go is going to be less appealing to you and I invite you to consider other options. If you do have that problem, you either A: already know how big a problem that is and will tend to appreciate some of what Go has to say, even if you don't quite agree with the solution or B: in another five or six years will fit into A.
But the command line interface is the fundamental user-facing piece of a command-line program. If it's not intuitive and clean, the usability of the whole program suffers. The internal API is secondary (this is why I didn't try to "improve" anything here -- go's flag package is good enough and I'm not trying to reinvent the wheel). It's all about a good experience for users, and I don't think Go's stdlib flag package provides that.
Commandline programs written in Go shouldn't feel like idiomatic Go programs, they should feel like idiomatic commandline programs.
It's interesting that you acknowledge that struggle but settle on what I'd consider to be a poor choice for your users. Potato/potahto, I guess.
[1] Whether you like it or not, GNU/BSD-style getopt/getopt_long is the standard these days, by a wide margin.
Also: getopt is pretty easy to write (if you're going for strict Unix style, you're presumably not looking for a featureful getopt), and that interface style is abused far more often than it's deployed gracefully. Some of the worst Unix command line interfaces† are the product of getopt, or the belief that there's an idiomatic Unix argument handling convention and that the idiom is getopt.
If your argument parsing needs are complicated, you should break out an actual parser.
Finally: I see the benefit of having an auto-generated usage message that is better than a bad auto-generated usage message, but if you really want artisanal command line tools, shouldn't you be writing artisanal usage messages? C programmers as a rule didn't count on the library to generate the usage message for them.
† See for instance nmap
You touch on the auto-generated usage message: personally I think the one that Go's flag lib generates is terrible. I agree wholeheartedly with the author. C programmers indeed don't count on libc to generate the usage mechanism (though, frankly, if it did, I certainly wouldn't mind), but we're talking about Go programmers here. And if idiomatic Go is to use the flag library, then you get an auto-generated message. And it's bad.
I think the other part of the author's gripe is just as (if not more) important: flag has decided to do away with the difference between '-' and '--' options, something everyone has come to expect as standard. In the Go model, it's weird to have multiple options for the same thing. For frequent users, typing "rm -rf" is better and easier than "rm -r -f", which is still better than (what I would think Go would encourage) "rm --recursive --force". The short options (and ability to chain them without a dash per option) is great for command-line users, and the long options are great for scripts where you want clarity and self-documentation.
Being able to make up complicated parsing rules is potentially another problem, sure, but I don't think it's as important to users of a Go program as either of the above issues that the author raises.
(For the record, I absolutely love Python's argparse. I think they've settled on a fair compromise between giving a lot of power, but making the simple cases simple, while providing behavior and UI that most users of my programs would expect. Not perfect, but I've been happier with it than with any other parsing library. Of course, implementing an API like that in a non-dynamic language that doesn't support optional, named function arguments might be difficult or impossible.)
But why should you have to do extra work to override a default that's sub-optimal from a standardization and user-familiarity standpoint, and is trivial to fix "upstream" where everyone should benefit?
Could you elaborate on why you believe flag is preferable to something like what I've described? (I'm coming from Ruby-land where Trollop basically does that.) I'm struggling to understand what's preferable about needing to roll your own parser and write documentation when you could just describe your program and have the parser/documentation auto-generated.
But when your needs get more complicated, when the information being communicated on the command line starts getting more dense, the concept "flags" becomes an obstacle. Far too many programs keep leaning on flags far past the point where the abstraction breaks down. You start to feel like you're writing small programs in the confining language of command line flags.
A great example I think is "tcpdump", because it exhibits both behaviors: it has a hideously overcomplicated getopt configurator and a relatively graceful parsed language for describing capture filters (those filters being the most important option given to tcpdump). You can see how tcpdump would be better off if most of the flags were hoisted into that filter language.
(They can't easily be, of course, because that filter language is actually parsed --- and then compiled --- by libpcap, not tcpdump).
It's OK for the standard library to satisfice or even constrain flag parsing, because programs that need "complicated flag parsing" are better served by something more sophisticated than flags anyways. Super-awesome flag libraries just trick programmers into building bad UX.
A lot of programs would also be better off if they reduced the numbers of flags they had, and made the "value" of those flags more sophisticated. In other words: taking some of the weight off the flag parser.
Most programs aren't tcpdump. Most programs don't have a lot of flags. But most programs that do have a lot of flags would probably be better off with something other than flags as their command line idiom. "tar" springs to mind.
It's hard for me to look at my zshrc and see all the aliases with their -fNqC's and -avz's and think "this is a good command line user experience". Many of these flaggy CLI programs have terrible UX.
There's maybe a "Gettysburg Address In Powerpoint"-style satire to be done of, say, awk commands expressed entirely in flags.
Check out an example: https://github.com/politician/azb.go/blob/master/src/cmd/azb...
[0] http://utcc.utoronto.ca/~cks/space/blog/programming/GoGetopt...
I'm glad for the recent changes from the flag package which has also been annoying me previously ;)
When i type blah -h or --help i expect help to pop up. I expect it to tell me about USAGE with [optional] and <required>. I expect the same spacing and readability as the vast majority (which is basically dictated by GNU getops.
Else, its a pain. Heck, even git doesn't follow this and as a result is a little more painful that needs to be.
Imagine a GUI program with every window having a different close, resize, etc button/menu. Hell.
I do wish they would improve the included flags package though. There are lots of nice third party packages, but having a proper one in the standard library would've been better.
But I think it serves as an excellent reminder that, for a cmdline program, the flags and arguments are the UI, they impact how people perceive your software and how productive they will be with it.