C Posix-compliant argument parsing in 42 LoC, inspired by Duff's device | Hacker News Reader

C Posix-compliant argument parsing in 42 LoC, inspired by Duff's device

github.com

81 pointscamel-cdr52 comments

This uses switch abuse to handle short and long arguments in one place:

    ...
    else if (ARG_LONG("option")) case 'o': {
    } ...

The idea was to leave all the decisions to the library user, so there are no automatic help pages and error messages. The library essentially just gives you a new language primitive to work with.

Example usage (also included in the file):

    // assumes argv and argc exist

    ARG_BEGIN {
            if (0) {
                    case 'a': a = 1; ARG_FLAG(); break;
                    case 'b': b = 1; ARG_FLAG(); break;
                    case 'c': c = 1; ARG_FLAG(); break;
                    case '\0': readstdin = 1; break;
            } else if (ARG_LONG("reverse")) case 'r': {
                    reverse = 1;
                    ARG_FLAG();
            } else if (ARG_LONG("input")) case 'i': {
                    input = ARG_VAL();
            } else if (ARG_LONG("output")) case 'o': {
                    output = ARG_VAL();
            } else if (ARG_LONG("help")) case 'h': case '?': {
                    printf("Usage: %s [OPTION...] [STRING...]\n", argv0);
                    puts("Example usage of arg.h\n");
                    puts("Options:");
                    puts("  -a,                set a to true");
                    puts("  -b,                set a to true");
                    puts("  -c,                set a to true");
                    puts("  -r, --reverse      set reverse to true");
                    puts("  -i, --input=STR    set input string to STR");
                    puts("  -o, --output=STR   set output string to STR");
                    puts("  -h, --help         display this help and exit");
                    return EXIT_SUCCESS;
            } else { default:
                    fprintf(stderr,
                            "%s: invalid option '%s'\n"
                            "Try '%s --help' for more information.\n",
                            argv0, *argv, argv0);
                    return EXIT_FAILURE;
            }
    } ARG_END;
   
    // argv and argc now hold the non-option arguments

This invokes UB, unfortunately. It works where it works.

ISO C says that:

"The parameters argc and argv and the strings pointed to by the argv array shall be modifiable by the program, and retain their last-stored values between program startup and program termination."

Thus within the boundaries of defined ISO C, you can do this:

  argv++;       // argv pointer to array element modifiable

and you can do this:

  argv[0][0]++; // the string pointed to by the array is modifiable.

but not this:

  argv[0]++;    // array itself (the pointers) not required to be modifiable

I seem to recall finding a problem like this in the K&R2 book also, probably close to thirty years ago. I don't have the book now, but the page number 117 seems to be coming up in my dim memory.

I'm skeptical there's a single C compiler out there that makes the first two behave as expected but not the third. In fact, I wouldn't be surprised if the real oversight is that the standard meant to say all three, or that the authors thought it implied.

It's not necessarily up to a compiler, or its program setup stub code, but possibly up to the host environment.

Of course argv is mutable since it's a local variable/param. But what if the pointers are placed, by the OS, into read-only VM pages? Begrudgingly, the string data is made writable though, to conform to the standard.

Please name one platform where this is the case.

Another thing to consider is that the pointers may be writable, but changing them causes harm. E.g. suppose argv[] is prepared by a function which dynamically allocates each pointer, and then the run-time calls free on it when the program exits. According to ISO C, that would be conforming.

I don't know of any platform where there are any ill effects. But the number of platforms out there with C implementations is much larger than my direct experience. Maybe this is just some historic baggage that is overdue for a revision. It was discussed in 1998, but no changes were accepted at the time.

In most coding situations, I probably wouldn't care, the same way I don't care about, say, platforms where for some reason you cannot compare pointers to distinct objects using inequality (another thing not required by ISO C). It's better to break the rules knowingly, though. You want to be badass, not dumbass.

Oh, wow thanks. I made sure to check the strings are modifiable, but that *argv may not be modifiable never occurred to me, I suppose this is an easy enough fix, I'll look into it later.

Edit: I just saw this SO comment [0]:

> The standard calls argv consistently an array. Modifying an array refers to modifying its members. Thus, it seems obvious that the wording in the standard refers to modifying the members in the argv array, when it states that argv is modifiable.

It is still a mater of interpretation so you shouldn't depend on it.

[0] https://stackoverflow.com/questions/35105918/are-the-pointer...

I don't see the interpretation issue. The "parameters argc and argv" are the scalar, local variables. "argv array" is the object that "parameter argv" points to; it is not a parameter.

The modifiability of argv was the subject of at three items in 1998 in meeting n849.

https://www.open-std.org/jtc1/sc22/wg14/www/docs/n849.htm

Someone proposed that clarifying text be added saying "Whether or not the pointers of the argv array shall be modifiable by the program is implementation-defined. If they are modifiable, they shall retain their last-stored values between program startup and program termination." This was rejected due to insufficient interest.

It was proposed that, since of course parameters are modifiable, the sentence in question be shortened to "The strings pointed to by the argv array shall be modifiable by the program, and retain their last-stored values between program startup and program termination", omitting the mention of "parameters argc and argv". This was discussed, but not accepted.

The question was raised whether the pointers are modifiable. Response was: "This is currently implictly unspecified and the committee has chosen to leave it that way."

Another option with similar levels of macro (ab)use is my "magic getopt" which lets you write code like this:

        const char * ch;
        while ((ch = GETOPT(argc, argv)) != NULL) {
                GETOPT_SWITCH(ch) {
                GETOPT_OPT("-a"):
                        aflag = 1;
                        break;
                GETOPT_OPTARG("--bar"):
                        printf("bar: %s\n", optarg);
                        break;
                GETOPT_OPTARG("-f"):
                        printf("foo: %s\n", optarg);
                        break;
                GETOPT_MISSING_ARG:
                        printf("missing argument to %s\n", ch);
                        /* FALLTHROUGH */
                GETOPT_DEFAULT:
                        usage();
                }
        }

switch options falling through doesn't seem like magic to me. Rather, a common practice.

That's not the magic part...

I'm intrigued. Is there somewhere where we could find the full definition of these macros?

https://github.com/Tarsnap/libcperciva/blob/master/util/geto...

BSD licensed, of course.

Very neat. Do you implement option parsing in "magic getopt" or can you (somehow?) handle setting up the option string arguments used by the more familiar variants in getopt(3)?

All the option parsing is done in functions called by the macros. The "getopt string" isn't needed since the GETOPT_OPT() and GETOPT_OPTARG() "case statements" convey the same information (the set of valid options).

I appreciate the macromancy, but it's not very readable. I'm sure it's very efficient though.

Perfect for the all-too-common situation where your performance bottleneck is in command line argument parsing...

don't people just use https://www.gnu.org/software/libc/manual/html_node/Getopt.ht... , rather than reinvent the wheel?

I realized that I essentially never use a "switch" statement. It seems like a control-flow construct that made sense in the 1970s to help the compiler generate jump tables, but doesn't seem particularly useful now. Moreover, it seems error-prone with accidental fall-through. And doing "fancy" things with it makes code very hard to read. Does anyone else find "switch" kind of redundant?

I use switch statements all the time for state-machines. There are linters that will warn on fallthrough; you should use one of them.

I still find switch statements easier on the eye than a chain of if-else-if-...else.

switch statements are better for certain cases since the compiler can warn if you aren't handling an enum variant within it.

I typically use them with more than 2 or 3 options as I find them much easier to read.

In addition, the compiler/vm typically can optimize them better than long if chains, depending on implementation, if that matters to you

If you don't need long options and don't want to use getopt (for whatever misguided reason) then I wrote a (rare) blog post about doing this without any macro abuse: https://the-tk.com/post/2021/07/29/option-parsing-on-a-budge... .

For the initiated, Duff's Device is a technique for manually implementing loop unrolling.

https://en.wikipedia.org/wiki/Duff%27s_device

(uninitiated. The initiated would already know what it is.)

Thanks for the typo correction.

PSA: Read the "simplified explanation" near the bottom of the wiki page (https://en.wikipedia.org/wiki/Duff%27s_device#Simplified_exp...) before looking at the actual code. I always found Duff's device confusing to understand (i.e. walk-through in my mind) and this section was what made it click.

How this code can enter the "if (0)" part?

A switch can jump inside an if body that contains case labels, causing the condition not to be evaluated.

C switch/case semantics are... not the most obvious.

Because there's really a switch statement (hidden behind a macro) that will jump to labels within that block.

The fact that the if condition is false means it won't just run the whole block straight through but you can still jump to a label in it. A goto statement would also allow you to jump into an otherwise-unreachable block.

ARG_BEGIN opens a switch statement. So it can enter any of the cases, or it'll jump to the else if part.

Is there any reason this is not written as `switch(..) { <case statements>; default: <else statements> }`. I.e is there a reason for `if(0)`?

The whole `if` portion is for if the second character of the argument (after the initial `-`) is a second `-` [0].

In that case, the switch jumped to `case '-':` hidden within the macro, and the if statement is about processing long arguments (like `--help`). arguments only available as a short flag should never get executed as part of a short flag, so putting them inside the if(0) case is an easy option.

An alternative without if(0) of any variety would be:

    ARG_BEGIN {
            if (ARG_LONG("reverse")) case 'r': {
                    reverse = 1;
                    ARG_FLAG();
            } else if (ARG_LONG("input")) case 'i': {
                    input = ARG_VAL();
            } else if (ARG_LONG("output")) case 'o': {
                    output = ARG_VAL();
            } else if (ARG_LONG("help")) case 'h': case '?': {
                    printf("Usage: %s [OPTION...] [STRING...]\n", argv0);
                    puts("Example usage of arg.h\n");
                    puts("Options:");
                    puts("  -a,                set a to true");
                    puts("  -b,                set a to true");
                    puts("  -c,                set a to true");
                    puts("  -r, --reverse      set reverse to true");
                    puts("  -i, --input=STR    set input string to STR");
                    puts("  -o, --output=STR   set output string to STR");
                    puts("  -h, --help         display this help and exit");
                    return EXIT_SUCCESS;
            } else { default:
                    fprintf(stderr,
                            "%s: invalid option '%s'\n"
                            "Try '%s --help' for more information.\n",
                            argv0, *argv, argv0);
                    return EXIT_FAILURE;
            }
            break;
            case 'a': a = 1; ARG_FLAG(); break;
            case 'b': b = 1; ARG_FLAG(); break;
            case 'c': c = 1; ARG_FLAG(); break;
            case '\0': readstdin = 1; break;
    } ARG_END;

The downside is that those few short flag only arguments no longer line up nicely with the others, and they come after the default, which could be a little bit confusing.

Footnote: [0] Except if that second dash is the last character of the argument, in which case -- is the end of flags marker, meaning any further arguments that begin with `-` are just funky positional paramaters

The `switch` statement in `ARG_BEGIN`.

The way "switch" works in C is very weird. It behaves more like a "goto", where each "case ...:" is a label to which it can jump.

So when the character is '-', it starts just before the "if (0)" (this part is hidden within the ARG_BEGIN macro), and as you noted, it will never enter that block. However, when the character is 'a', 'b', 'c', or nothing (the end-of-string marker), it will jump directly to the corresponding "case ...:" label, even though it's within that "unreachable" block.

What does posix compliant mean in this context?

It (edit: claims to) conforms to the POSIX spec for command line arguments

https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...

That document does not describe long options. I guess that can be treated as a conforming extension.

The C implementation does not seem to have support for optional option arguments.

The C implementation also does not seem to support "--" as a delimiter for option arguments, although that's only a guideline.

So I guess it can be used to implement POSIX compliant interfaces that don't use optional option arguments.

edit: supporting "--" as delimiter might not be a guideline:

The utilities in the Shell and Utilities volume of POSIX.1-2017 that claim conformance to these guidelines shall conform completely to these guidelines as if these guidelines contained the term "shall" instead of "should".

edit2: Seems like I was completely wrong about supporting "--", when tried out it does seem to support it. One weird corner cases is the call `<prog> -- -`, where "-" isn't interpreted as enabling stdin, but treated as a regular positional argument. `echo asd | cat -- -` reads from stdin with GNU cat.

On a closer read, the comment mentions plan9's arg(3):

https://9fans.github.io/plan9port/man/man3/arg.html

Generally, I think it's pretty normal to interpret "POSIX compliant" as a minimum requirement, and then extend that to your own whims as long as you don't break the POSIX part.

"--" is a GNU originated thing afaik.

Maybe, but it's part of POSIX now. Anyway, I misread the code, and it's actually supported.

> So I guess it can be used to implement POSIX compliant interfaces that don't use optional option arguments.

Yes, that was my intention.

In the context of option parsing, "POSIX compliant" makes me think of the semantics of getopt(3): https://pubs.opengroup.org/onlinepubs/9699919799/functions/g... Notably, the utility conventions and even the guidelines (guideline 14 presumes non-compliance with guideline 7) support optional option-arguments, while the bundling semantics of getopt preclude optional option-arguments. Utilities, including POSIX-defined utilities, which have optional option-arguments cannot use getopt.

This is off topic, but your stretchy buffer does one of my pet peeves:

    (a)->at = realloc((a)->at, (a)->_cap * sizeof *(a)->at))

When realloc fails, the old value of (a)->at will be leaked.

And memory allocation failure also leads to null pointer dereference in this code.

Ah, my way of dealing with memory allocation failure is to write a xmalloc/xcalloc/xrealloc that exit on error. So the idea would be that the library users can just:

    #unded malloc
    #define malloc xmalloc
    ...

I suppose I could make this the default behavior, and make it overwritable.

Doesn't this break things like Valgrind?

Firstly, an xmalloc will call malloc and add error checks. So malloc is still called.

But more fundamentally, Valgrind hooks into the allocator (and the program at large) much more deeply. You could write your own allocator (many people deploy something other than libc's) and it would still figure it out.

Compliant

Still, complaint argument parsing is an important and undervalued skill, POSIX or not.

I clicked through to the link before reading any comments:

"In C? I wonder if it's English only?"

Then proceeded to become extremely confused:

"How is this going to help me parse a complaint? It looks like arg parsing..."

Whoops, is it possible to edit the title?

Reply on news.ycombinator.com