Hacker News Reader

Comment by acmj | Hacker News Reader

In general, command-line argument parsers should just follow the GNU style. No more, no less. Deviations confuse users as it is not immediately obvious to them what rules a parser is imposing.

> options can have multiple values: -a 1 2 3 means that a is an array/slice/struct of three numbers of value [1,2,3]

Allowing multiple values is inconsistent because you can't tell in "./cmd -a 1 2 3" whether 2 and 3 are positional arguments or arguments for -a. This is not a GNU style. The GNU way is "./cmd -a 1 -a 2 -a 3" (or "./cmd -a 1,2,3"). This package supports that, which is good.

> option values can be separated by a space, equal sign, or nothing: -a1 -a=1 -a 1 are all equal

"--a=1" is a GNU style but "-a=1" is not. This is a minor issue, though.

Also, does this package support "--"? Everything following "--" should be treated as positional arguments.

View on HN

I disagree with it being a minor issue. If I write a shell script around a program that accepts GNU-style arguments, I expect the following to be correct:

    ./cmd -a"$USER_CONTROLLED_DATA"

A program using this package would break that assumption, introducing a bug where this user-controlled data cannot start with an '='.

I researched this for my own argument parser (https://github.com/blyxxyz/lexopt/issues/13) and concluded that it's a minor issue.

This syntax is supported by argparse and clap, the most popular argument parsers for Python and Rust respectively, and it seems to have caused almost no problems for them. It's a problem for the uutils implementation of cut, since `cut -d=` is common, but that's the only instance I could find after a long time scouring search engines and bug trackers and asking for examples.

If anyone does know of other examples or other places this has been discussed I'd love to hear it though, maybe I just haven't found them.

(Also, the more reliable way to write this in general is `-a "$USER_CONTROLLED_DATA"`, since that'll behave correctly if $USER_CONTROLLED_DATA is empty. As will `-a="$USER_CONTROLLED_DATA"` if you know the command supports it.)

In the Gentoo world, sometimes you need to give an exact package name which looks like `=net-misc/foo-0.1.2-r1`. The exact match has to start with the '='.

I think short options taking a value in the same argv (i.e. `-o=1` stuff) isn't a GNUism mostly because it's backwards-incompatible with POSIX. `=` is a valid getopt option character, `chmod` uses it.

That said, I think? 'nloomans means for USER_CONTROLLED_DATA to be a set of short flags, not flag values, as in:

    root@08e9950d5bfd:/# export USER_CONTROLLED_DATA=lh 
    root@08e9950d5bfd:/# ls -a"$USER_CONTROLLED_DATA"
    total 56K
    drwxr-xr-x   1 root root 4.0K Mar 23 16:51 .
    drwxr-xr-x   1 root root 4.0K Mar 23 16:51 ..
    [...]

Not that I've seen this in the wild before. But everyone's use of bash is a unique personal hell anyway.

No I did mean flag values. Allowing user-controlled short flags is horrifying.

Ha! I just said that. :-)

Anyway, one other alternative for the `cut` situation is to allow either ':' or '=' to optionally separate the key and the value. Then you can say `cut -d:=` or `cut -d=:` if you wanted to use either one. This is what https://github.com/c-blake/cligen does (for Nim, not Go).

The problem is existing shell scripts and muscle memory and command histories. `cut -d=` has always worked and works on all the other implementations so it should keep working if you switch to uutils.

Literally any short option key that takes a string or a char that could legitimately start with '=' has the problem, though, not just `cut`. The '=' will be "eaten" by one tool and left in another. But you know that. You write a bit as if we disagree, but I don't see any real point of contention. :-) Also, uutils has a very strict "drop-in" agenda. So, as another e.g., if you want `cp -t=foo` or `cp -S=foo` to work the same, you're going to have trouble if that '=' is eaten.

So, in this case that would seem to imply a problem for any utilities with options taking strings-or-chars, not merely `cut -d`. If uutils really wants to be strictly drop-in compatible, they may well need to roll their own option parser or twist the arm of whoever's they use to provide a mode for them.

In the more general case, "cross compatibility" may just always be limited by the reality that people just disagree on this stuff "more than they seem to think they do" (at least in my experience) and definitely more than they wish they did. I surveyed my /usr/bin once and like half of thousands of commands did not work with --help (yes, running that took some confidence in backups! but anyone could replicate on a throwaway VM or something). Consistency is nice, but consistency with what? -l=foo is consistent with --long=foo, but not (some, but not other) historical things.

I'm not sure there will ever will be a world in which you don't need to know which PLang/CL toolkit was used to make a CLI utility if you really want to know its syntax. The article's lib is going its own way from the Go stdlib. POSIX is pretty darn calcified. A 15 year old Python stdlib thing is unlikely to ever change in this regard. Python also allows "--beg" for "--beginning-long-option" if nothing else starts with "--beg" even back in its optparse days and that also tends to be controversial. cligen tools actually provide a --help-syntax. Maybe something like that could take off?

I can think of a lot of cases where it theoretically could be a problem, but `cut -d=` is the only one I've found so far where an end user ran into trouble because of this ambiguity, and I think it's the only one for which uutils bothers implementing a workaround. That's why I give it special attention.

> You write a bit as if we disagree, but I don't see any real point of contention. :-)

The `cut -d:=` spelling solves a different problem than the one I meant (and the one you're now talking about). But we're mostly on the same page!

I'm sad docopt (http://docopt.org/ ) didn't get more uptake because I'm a fan of self-documenting code.

It is safer to just put in the space (much like you put in the quotes to be safe). Python's argparse will also accept but not require an `=` separator (maybe optparse, too - I haven't checked that one).

It's only "safer" because argp has this particular bug. It's safer for argp (or python's argument parser, for that matter) to not have surprising buggy features like this.

While it may go against your personal expectations, I believe at least the Python variant is intentional not an accidental/unintended "bug".

"Surprise" is observer-relative. From a blank slate, if --long=val and --long val both work, then why not -s=val and -sval and -s val for a short -s?

So, I think the right word here is "disagreement" (perhaps about "which consistency - history vs. internal"), not throwing shade by simply declaring/asserting it a "bug".

Unfortunately a lot of people nowadays don't seem to notice or care and you even get weird arguments like the one about "Go style" vs "GNU style" below. I've also heard arguments about making it more "user friendly" (at the cost of any power user wanting to re-write your tool).

I wouldn't mind if GNU style was consistently extended to allow annotated required arguments. But that's about it.

That being said. As a rule, if your command line utility has enough positional arguments that you're forgetting which one is which, it's a badly designed command line interface. Most often it's because you're doing too much, not settling on sensible defaults for things which you shouldn't need to specify every time, or just doing something outright weird.

IME commands get there by expressing complexity that belongs in a full configuration file on the command line (as well / instead).

It's nice if things are simple enough that a hand-full or two of flags are sufficient. However more complex programs that do more or substantially different things from traditional bytestream filter programs often do need proper configuration files.

Just to clarify, the documentation was a bit outdated, but `./cmd -a 1 2 3` would not mean an array of `[1,2,3]` but exactly as you mentioned: option `a` with two positional arguments. And yes! The `--` is supported :-)

Are you saying this because you think the GNU standard is particularly well thought out, or because it's so common that deviations are confusing in general?

I'm so used to the GNU conventions that I'm not really aware of what the alternatives are or what their merits are.

The most important part is to have a standard, which doesn't need to be perfect. On argument parsing, I actually think the GNU way is the best so far. I have seen various deviations from GNU and I personally regard all of them inferior.

GNU became the de-facto standard at least 20 years ago

No, command tools should follow the POSIX standards, not GNU.

POSIX doesnt even define long opts. I conciously dont follow POSIX these days. The standard will adapt once enough pressure has built up as they mostly document existing things instead of innovate. So I need to apply pressure.

You're almost right. No command tools should follow the POSIX standard. A standard which all but forbids long options deserves only scorn.

POSIX doesn't even provide any utilities to write long opts and it also doesn't even define any long opts for basically any of its commands, so by default it is barely usable in maintainable scripts.

Disagree. Go's style may not be GNU style, but it's consistent in the Go community. GNU style is not All That and a Bag of Chips.

> it's consistent in the Go community.

Except it's not? There are tons of flag packages in real use, because the stdlib one kind of sucks. Even the "go" tool itself works around some of its limitations (things like "go test ./... -v" won't work out of the box, since it will stop parsing flags at the first non-flag, so the go command reorders os.Args before sending it off to the flag package).

Things like "-a 1 2 3" are not standard "Go style" at all. I've never seen that.

> because the stdlib one kind of sucks

It doesn’t. It’s simpler, which makes it not suck in my books.

All the stuff about short vs long, grouping short flags together, allowing the last flag of a group of short flags to take an argument, allowing a flag to take multiple values, etc. is cool, but confusing and quite hard to do right.

In Go it’s just -flag[=value].

You can't expect the user to remember upfront what language the tool they're using was written in and context switch the style they're expressing their intents in based on that. The Go community has consistently shown it's not very good at making well-thought out decisions. GNU is obviously the better approach here.

> The Go community has consistently shown it's not very good at making well-thought out decisions.

Please, do name one language that has.

I think you mean "inconsistent at making good decisions", which one can expect to be quite common, but GP probably meant "consistently making bad decisions", which is different.

I'm implying that a statement pointing at and deriding A for doing X is useless or misleading, if every alternative to A does X, arguably even more so. The precise interpretation of a given X does not matter.

As a heavy user of Go tooling, I still really dislike Gos style.

Agree. It was an unnecessary thing to do.

you two should get together and propose a better solution, with a working implementation they could adapt if needed, and integrate.

talk is cheap. (I'm not saying that you're full of hot air, I'm saying that if you want to make Go's flag package better, make a proposal with good rationale behind it, include a good implementation, and see what the community says.)

Go's flag package is covered by the backwards compatibility promise, and no large-scale change is going to be accepted. Pushing proposals to it is a waste of time.

But it doesn't need to be changed. Nothing stops anyone from using any of the several argument parsers for Go. The one included in the standard library is just that and nothing more: included in the standard library. It is not privileged in any manner (beyond that brute fact), it has no access to any sort of internal standard-library only functionality that gives it access to performance external packages can't have, it isn't particularly integrated forcibly into anything else included with Go, it is not mandatory, it is not even something the community deeply believes must be used and anyone who uses alternate mechanisms is violating any sort of nebulous community standard. It's just a package.

This is one of those cases where people often rather casually say "I don't like how Go handles X", or, indeed, any language with a standard library (this is not a Go-specific problem) when they really ought to say something more like "I don't like how the standard library flags package in Go handles X". But it's not the language doing it, it's the library. For a non-Go example that I've encountered in the wild, it isn't really appropriate to say "I don't like Python's GUI", referring to the built-in Tkinter binding. It's just a library. Python's got a dozen other choices, including direct bindings to every major toolkit in every major OS. It's not "Python's GUI", it's just one that ships in the standard library, not the sum total of all of Python's GUI capability.

Could not disagree more. Everything that affects your use of a language is that language - the standard library, the ecosystem, the community, the documentation, the tooling, every single thing.

The standard library is special. It sets the blessed way to do X, e.g. Context, the Writer and Reader interfaces, etc. Claiming that you may as well write your own is like claiming you can just fork Linux if you disagree with its direction. Good luck with that.

Bringing up Python is particularly unfortunate since its packaging story sucks, so whether a package is included in its stdlib or not really matters.

Sure, argv parsing is an isolated piece of functionality, so in that specific case it doesn't really matter what package you use, but the sentiment is incredibly wrong in general.

You imagined that I was saying that the existing flag stuff in the standard library should be replaced, I never suggested that.

You can add better things without breaking compatibility promises. See: many, many features added since 1.0. New IP address types, slog, etc.

People are not looking to "add" to the library. They don't like it at all and want to completely stop it from doing what it does. There is no proposal process that is going to make them happy.

Fortunately, that is not a prerequisite for their happiness.

The standard library isn’t going to change because of Gos backwards compatibility promise. Which is a good thing.

People can import their own flag parsing libraries, hence the HN submission we are discussing at the moment.

As for “talk is cheap” comment, I’ve written more open source code for working with software in the command line than most people. And that includes tools that parse other CLI tools and man pages to provide autocompletion hints. But that still doesnt mean that Go should make a breaking change to their standard library based on my personal preferences.

I didn’t say that you should aim to replace the existing package. You imagined that as a requirement.

How did log/slog appear if the Go standard library already has a logger and a backwards compatibility promise? Because you can add stuff without breaking backwards compatibility.

Go’s backwards compatibility promise is good in the short term, but if it has no expiration, that promise will be the rope wrapped around Go’s neck as it is pushed off a small platform and is left to dangle as the crowd watches.

Do you think that backward compatibility promise will still be upheld in 500 years? If you think not, then you agree that there is a line somewhere between 0 and 500 years and that on the other side of that line that promise makes no sense.

What about 50 years? Do you think in 50 years we’ll still have the same flag package in the standard library that we have today? I don’t. I certainly hope we don’t.

The more one clings to that backward compatibility promise, the more the backward compatibility promise strangles you.

I hope we incorporate lessons learned from Go 1 into some new version of Go, whatever its name is. I certainly DO NOT hope that we ignore what we’ve learned and fail to act because of a promise to never abandon the mistakes that have been made.

The original points I make are still just as valid. Your counter, while not technically incorrect, doesn’t change the nature of the problem nor change my mind that a Go proposal is the right way to focus my energy at this point in time. Particularly when I’m already focused on several other open source projects in the same problem domain right now.

Time is a finite resource and there are already other packages out there that satisfy the problem, so why add to the noise?

One could flip the argument and say: if you’re so convinced that a proposal is the right course of action, then why don’t you create one yourself?

> if you’re so convinced that a proposal is the right course of action, then why don’t you create one yourself?

because I'm not annoyed by the existing flag package, and the people I was replying to are, that's why. that's why I suggested that they submit a proposal in the first place.

this is apparently a contest to see who can misunderstand me the fastest and you're all winners in this race.

I think you’re the one who’s misunderstanding things here.

We aren’t annoyed. We said we dislike the one in standard library.

That’s also precisely why alternative packages exist. Like the one you’re commenting on right now. No proposal is needed.

I also admire your optimism regarding the ease in which people can push a proposal in Golang. ;) Gos famously hard in that regard.

you said you "really dislike gos style".

there is no better person to produce a proposal than the person who best understands why it isn't sufficient or is wrong in one or more ways.

I didn't say it was easy to guide a proposal from the proposal stage through to acceptance, or that I was optimistic about doing it. (you think I am misunderstanding you?) I merely feel like the people who like something the least are probably the most qualified to produce a quality proposal which has a chance at improving the landscape. that's all.

By the way, if i haven't said something outright, I did not infer that thing. Don't read between my lines, there's nothing there.

> you said you "really dislike gos style".

Exactly. Dislike != annoyed

I can use a different package so what’s there to be annoyed about?

> there is no better person to produce a proposal than the person who best understands why it isn't sufficient or is wrong in one or more ways.

Dislike != insufficient

It’s a personal preference not a technical criticism.

> By the way, if i haven't said something outright, I did not infer that thing. Don't read between my lines, there's nothing there.

If you’re going to play the “you’ve misunderstood me” card then at least make sure you’ve understood people correctly yourself.

> I didn't say it was easy to guide a proposal from the proposal stage through to acceptance, or that I was optimistic about doing it

Changing the way the flag package would need to be changed to make myself (and others) happy couldn’t be done in a backwards compatible way. Thus it would have to be part of Go v2.0, which could be a decade away or might never happen.

And since all that the change would bring is an alternative, more GNU, way of parsing flags, it makes this hypothetical proposal a really low bar for improvement (unlike your other examples) thus this hypothetical proposal is unlikely to ever approved even if a hypothetical v2 of Go were to ever be released.

The whole premise of this changing is so remote that even suggesting I write a proposal in the first place is a massively optimistic take.

And for the record, I fully support Go not changing their behaviour here. Keeping the flags package as-is is the right thing for Go to do. So I wouldn’t even support my hypothetical proposal here even though I don’t personally like the existing flags package.

This might sound too nuanced for HN, but you can dislike something while still supporting it’s existence. Change here, in my opinion, would be worse than the status quo.

When I use a command-line tool, I don't know and don't care if it is written in Go or not. I just want to use it the same way as most of the other traditional Unix tools. The Go style gets in the way. We would have much more consistent CLI between tools if Go had just followed the GNU style at the beginning. The same can be said to many other languages like Nim that want to reinvent command-line argument parsing with their standard libraries. https://xkcd.com/927/ came to mind.

Can’t you just use --long options everywhere?

For, like, 95% of the tools out there, I’m going to use --long every single time.

Anyway, I don’t even know the options that a program takes until I read the program docs. The program docs will tell me that the option is -host=localhost or --host=localhost or --bind-addr=localhost:8000.

The other 5% are tools like ls, cp, mv. As far as I care, ancient tools are the only ones permitted to have short options that combine into a single option, like old-school getopt. Maybe a few exceptions now and then.

I use short options when I'm typing at the command line, but for scripts I prefer the long options by a wide margin. It's just too painful to come back to script after some time and see a string of short options that looks like line noise barfed out of a 300 baud modem.

Reply on news.ycombinator.com