Hacker News Reader

When it comes to avoiding forks, the question is why do egrep, fgrep have to be shell scripts rather than interpret the invocation name (argv[0]) for switching behavior. Also, I find the original POSIX spec to lump everything into a single grep binary questionable since egrep, fgrep developed as extension and restriction, resp. of grep under a classic regexp discourse and it isn't clear at all that automaton construction for powerful egrep support has to live in the same binary as fgrep which isn't using automaton construction at all.

> When it comes to avoiding forks, the question is why do egrep, fgrep have to be shell scripts rather than interpret the invocation name (argv[0]) for switching behavior.

I don't know. I'd have taken the argv[0] approach personally. And it's the approach some other grep implementations have taken too (eg https://github.com/freebsd/freebsd-src/blob/main/usr.bin/gre...).

The argv[0] approach (to me) feels like a right kind of compromise.

Switching behavior according to argv[0] been used in Unix for decades. mv(1)/ln(1)/cp(1) historically are the same program, hardlinked.

Wasn’t rm also the same?

I vaguely recall it being little more than ‘mv $1 /dev/null’ under the hood. But this might be something I’m misremembering.

Yes. I made a mistake, I think cp(1) was separate.

The gnu coding standards has this to say: https://www.gnu.org/prep/standards/standards.html#index-beha...

Please don’t make the behavior of a utility depend on the name used to invoke it. It is useful sometimes to make a link to a utility with a different name, and that should not change what it does.

The next section provides some reasoning:

Providing valid information in argv[0] is a convention, not guaranteed. Well-behaved programs that launch other programs, such as shells, follow the convention; your code should follow it too, when launching other programs. But it is always possible to launch the program and give a nonsensical value in argv[0].

Thanks for the link, but I find that reasoning entirely unconvincing:

The ability to interpret argv[0] as a way to avoid exec is traded for hypothetical linking - but wherever linking is used (such as for idk a mail filter or other configurable executable) that's the place where a wrapper shell script could be used instead. And avoiding improper launching of egrep/fgrep from a program that doesn't bother to build argv[0] properly is also an (unlikely) non-use case that's better handled by fixing that offending program.

I find the reasoning convincing. It's the principle of least surprise. And a bunch of other principles. I don't want a program to care what it's executable name is, if it's being invoked with a relative path, absolute path, through a symlink or hardlink or anything else.

Of course, there are exceptions to the rule, like busybox. The point is to have uniform, predictable behavior across the ecosystem and not have every other program have its own weird idiosyncrasies.

That being said, the egrep/fgrep legacy is precisely such a case where it makes sense to make an exception. It's a decades-old legacy and grep isn't just any program, it's part of Using the Shell 101.

What makes the reasoning unconvincing for me is that it's simply wrong: argv[0] is not the name of the program being invoked. If it were, I would agree, but it's not. Rather, it's simply another argument whose default value is the name of the program being invoked, but you can pass something else in place of it if you want. Moreover, what I find surprising is seeing a knob that I can adjust, but which doesn't do anything. It's like having an argv[1] parameter that is always ignored. It's much better to make it do something that logically corresponds to its value, and the same goes when the index is 0.

> argv[0] is not the name of the program being invoked [...] it's simply another argument whose default value is the name of the program being invoked

That's your opinion. Another opinion would be that it's an interface contract with the calling program to pass the path to the program being invoked in argv[0]. This contract has been established by common practice and a corresponding expectation by most programs, even if it wasn't formally specified.

I'm not necessarily taking that position, but I can see convincing reasons to do so, and as such the reasoning in the article is also convincing.

That's not an opinion, that's a fact. The only thing that's an opinion in what I said is what the final decision should be, not the fact that it's a mutable argument. Whereas their stance is based on false premises to begin with. If their opinion was "we realize arg0 can be set explicitly but too many people wrongly assume it can't, so we will follow the crowd" that would make their argument more compelling.

The fact that it can physically be set to anything doesn't mean there isn't an implicit contract.

This isn't just "I can set it to something else if I go out of my way", this is "systems provide well-documented and standardized methods for setting this parameter to something else, and there are some widely used programs using precisely this feature." It's not common but that's exactly what default parameters are for: uncommon-yet-valid use cases. Yet you seem to treat your opinion that this is an implicit contract as somehow sufficient for establishing that it's a contract, despite what I just mentioned indicating otherwise. What contract can you point to that mandates argv[0] be set to the program name?

Established common practice.

The egrep use case, btw., wouldn't deviate from that. In fact, it would rely on argv[0] being truthfully set to the sym/hard link's name.

Not every established common practice implies a contract to follow that practice. It also matters what the reason for that practice is, and whom it's even relevant to. Like just because most people take the highway when driving from SF to LA that doesn't mean you're breaching some sort of contract by opting to take a side road.

There are two practices here. One is passing the program name to arg0. The other is never making some use of arg0. Both of these are established because they're the most convenient things to do by default, and because people rarely have a reason to deviate from them. That's it. Heck, if there was some contract to ignore arg0 then people wouldn't feel any contractual obligation to pass the program name to begin with, since it should be getting ignored anyway - that argument is pretty self defeating. Moreover, I think your position is effectively equivalent to saying "if you don't want a common practice to become a contract, then you must go out of your way and inconvenience yourself to deviate from that practice for absolutely no other reason than to make this very statement true", which is a rather bizarre (and inefficient) expectation from everyone around you.

Different behavior based on argv[0] was first brought to my attention when I discovered that /bin/sh was a symlink on some Linux systems. Bash has a Bourne shell compatibility mode.

Busybox is the forefront example of this in my mind. Busybox is a single binary that provides a ton of POSIX utilities (including sh, awk, ls, cp, nc, rm, gzip, and about 300 other common ones), and these can be invoked with `busybox $command`, or a symlink may be made from the command name to the `busybox` binary to automatically do the same. Many embedded Linux systems just use busybox and a ton of symlinks to act as the majority of the userland.

GNU does it their way, and Busybox does their own way too. Both have valid reasons for how things are set up. For users friendlinesr it's important then that they are as consistent as possible with it.

Bash actually has a POSIX compatibility mode.

There is a great deal in POSIX that was not in Bourne, native arithmetic expressions being the first to come to mind, then new-style command substitution.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...

All of the POSIX utility standards, including the grep variants, can be found in the URL's parent directory:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/

If this new GNU grep functionality becomes widely distasteful, I think that OpenBSD's grep tries to emulate GNU in a few ways, and could supplant it, becoming the system grep while "gnugrep" is relegated to the shadows.

It is also extremely expensive on Windows for a version of grep to be implemented as a shell script. Launched by xargs, the penalty will be severe.

The commercial platforms based upon GNU are in a marriage of convenience, and can easily pick and choose.

> Bash actually has a POSIX compatibility mode.

It's partial. A short example is `date&>FILE`.

[edited typo]

On a POSIX shell, `date` will run in the background and the date will be written to stdout (`date&`) and `FILE` will be created or truncated (`>FILE`). Using `bash --posix`, the date will be written to `FILE`, since the incompatible bashism `&>` still takes priority.

I agree that is ambiguous.

If I were writing such a script where I wanted to launch a background process, and then create/truncate a file, I myself would separate them:

  date &
  >FILE

Bash wouldn't mistake that, but a lot of shell scripts look like line noise and there are situations where bad form is required (quoted in an ssh, for example).

Obviously, people wanting the bash functionality in POSIX would:

  date > file 2>&1

Bash discourages the latter form in my man page, but anyone interested in portability knows how profoundly bad that advice is.

Yikes. This is why I do not use bash. Too complicated.

For Linux, I added tab autocomplete to dash and use that for both interactive and non-interactive shell instead of bash.

Saves keystrokes and bytes. No need to keep typing "#!/bin/sh" into every script.

You can compile dash with a line editing library, but it does not come in most packages.

I use libedit. I am used to NetBSD sh; dash is similar, derived from the same source. I wanted dash to behave more like NetBSD sh, so I changed it. Anyway, I am not a huge fan of "packages". (The problems described in this thread are one reason why.) Of course I use packaged binaries in some instances, but normally I only install them when needed, use them and then uninstall them when I am done. I prefer to compile the kernel and userland myself. That includes the shell. Simply compiling dash with libedit does not provide tabcompletion. Need to make some small changes.

I just checked OpenBSD, and I find that there are 6 total links in /usr/bin: egrep, fgrep, grep, zegrep, zfgrep, and zgrep.

The OpenBSD package is superior.

The GNU gzip package includes a zgrep shell script that is adapted from work by "Charles Levert <charles@comm.polymtl.ca>" - this is similarly adapted for bzgrep and xzgrep.

The OpenBSD implementation will have superior performance for zgrep, because it is native C.

  rebel$ ls -li /usr/bin/*grep 
  466612 -r-xr-xr-x  6 root  bin  31520 Apr 11  2022 /usr/bin/egrep
  466612 -r-xr-xr-x  6 root  bin  31520 Apr 11  2022 /usr/bin/fgrep
  466612 -r-xr-xr-x  6 root  bin  31520 Apr 11  2022 /usr/bin/grep
  466711 -r-xr-xr-x  2 root  bin  15288 Apr 11  2022 /usr/bin/pgrep
  466612 -r-xr-xr-x  6 root  bin  31520 Apr 11  2022 /usr/bin/zegrep
  466612 -r-xr-xr-x  6 root  bin  31520 Apr 11  2022 /usr/bin/zfgrep
  466612 -r-xr-xr-x  6 root  bin  31520 Apr 11  2022 /usr/bin/zgrep

> But it is always possible to launch the program and give a nonsensical value in argv[0].

Well, if you're being a smart@ss and doing that, don't expect it not to break later.

Putting a weird value there and hoping it won't break is as stupid as calling the program with wrong options and hoping it will work.

Exactly.

the name of an executable is like a global variable the instant it is used as data; it can be changed at any time by someone else and then behavior changes.

names are not data. don't make them data.

names are names.

Using argv[0] for dispatch has worked for 4+ decades. There is zero non-contrived evidence that it ever fails to work.

Further, there's 4+ decades of legacy here, and it's very rude to break backwards compatibility with something like this where there's no need, no security vulnerability that must be fixed by breaking backwards compatibility.

I'm not saying it doesn't work, I'm saying that relying on it being a certain value, or one of a set of values, is like relying on the value of a global variable.

we avoid that in software development-land whenever possible, and for good reason.

Your point is moot given that this works and has for decades, and you concede that point.

> we avoid that in software development-land whenever possible, and for good reason.

We also avoid breaking backwards compatibility unless it's really necessary. In this case, granting your premise for the sake of argument, the harm caused by this change exceeds the actual non-harm of the thing you're objecting to regardless of the theoretical badness of that thing.

So if you symlink grep to something else then you have to use `-E` for that, for the 99.999999% use case of the rest of the world egrep being a hardlink to grep would have worked just fine.

> I'd be surprised if that warning made its way to the mainstream distros.

This is the first time I'm hearing about the warning, but I thought I'd run `egrep` and `fgrep` on my Arch system. Sure enough, both result in warnings: `egrep: warning: egrep/fgrep is obsolescent; using grep -E/-F`

So it's made its way into Arch, at least. Though, like I said, this is the first time I'm hearing about this.

I use arch, and I had an alias for grep to expand to `egrep --color=always` in my interactive shell. I noticed real fast. Fortunately it's an easy fix to `grep -E --color=always`.

sidenote: I wish that grep would allow running `grep -E -F`, and have the `-F` flag override the `-E` flag, rather than giving an error about conflicting options, so that I could have an alias like this to make extended regex the default, but allow me to change it with a flag.

You could write a little function that does that

For me, that's the biggest takeaway:

Nothing is stopping users from maintaining the behavior they had previously.

It's being made explicit that the burden of maintaining that behavior is changing.

Yeah, this is a good point. This is basically just GNU saying that those old aliases are not something they want within the scope of grep itself.

What exactly do you think the burden of _not_ changing a shell script is?

I too was surprised to see these were shell scripts. I was expecting the grep/fgrep/egrep names to be hard links to the same executable that would check `argv[0]`, as the BSD implementation does.

More interesting than the commit diff is the brief discussion on the bug report from when this all happened a year ago, as referenced in the commit message: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=49996

Sigh. Time to retrain the old fingers again...

Thanks for sharing that. It was an interesting read. Particularly this comment:

> The irony... that one of our own tests used fgrep!

If ever there was an argument for fgrep becoming a pseudo-standard that should remain, that would have been it.

> though it has already been committed to git for more than a year now and this is the first I've heard people complain about the change.

This reasoning is flawed. I only noticed it a few months back on Arch. So between the time it takes the project to make a new release and it hitting Linux distros (Debian is slower and Ubuntu even moreso), don’t expect a flood of complaints to be time correlated with the change.

> This reasoning is flawed.

I wasn't making any reasoning. I was just making the observation that this change has been staged for a while now and that I'm surprised there hasn't been more noise about it before now.

> don’t expect a flood of complaints to be time correlated with the change.

I expect complaints to be correlated with whenever a blog/tweet/whatever moaning about the change happens to trend. I don't think it has much to do with when distros pick up the change because, as I also said, I expect distros to backport the old behavior. So I think the timing of any backlash is entirely dependent on the mood of the internet hive mind.

> I was just making the observation that this change has been staged for a while now

“There’s no point in acting surprised about it. All the planning charts and demolition orders have been on display at your local planning department in Alpha Centauri for 50 of your Earth years, so you’ve had plenty of time to lodge any formal complaint and it’s far too late to start making a fuss about it now. … What do you mean you’ve never been to Alpha Centauri? Oh, for heaven’s sake, mankind, it’s only four light years away, you know. I’m sorry, but if you can’t be bothered to take an interest in local affairs, that’s your own lookout. Energize the demolition beams.”

Again, you're completely missing the point of the comment you're replying to: average users don't pay attention to what's being staged upstream. You're only going to get the real flood of complaints when it actually gets pushed out and people's houses start getting demolished.

> Again, you're completely missing the point of the comment you're replying to

Actually you and the GP are the ones missing my point by obsessing over a throwaway comment about this commit being over a year old. I was literally making zero conclusions from that observation. You guys are reading far far far too much into that comment. You seem to be projecting your annoyance about this change onto me as if I’m defending and supporting this change, yet literally nothing I’ve posted has supported that claim.

> average users don't pay attention to what's being staged upstream.

I’m going to assume that you skipped over my point about how any outrage will come from blog/Twitter/etc posts going viral.

This has already landed on some distros and most people, rightly or wrongly, went “meh”. If that last 30 years of the internet has taught me anything, it’s that people get outraged by posts, not by software. And the fact that youre arguing over a meta-point like when people will get annoyed, rather than discussing the technology itself, really just confirms my 3 decades of observations.

In fact the only reason we are discussing this now is because someone blogged about it and it hasn’t even hit the distro they’re using; they know about it because they read another news article who found out about it from the release notes posted on the mailing list and the authors then went back and checked the commits! Literally nothing in that chain of discovery was via experiencing the change itself in their chosen distros.

So to be clear:

I am NOT suggesting that the commit being > 1 year old means GNU have a free pass to make a breaking change. Any conclusion like that you derive from my posts are a misinterpretation and not worth arguing over.

Now, can we move on to more interesting things?

The maintainers' fears about using argv[0] to select behavior are unfounded.

Anyone exec()'ing grep with an alternate argv[0] can just use (or not use) the -E/-F options to get the behavior they want.

It's very simple. There is just no good reason for egrep or fgrep to be shell scripts or wrapper programs of any kind. Even if the were a good reason for having a wrapper program, the wrapper program could be a very tiny C (or even assembly) program that just exec()s grep with -E or -F added, thus avoiding any additional fork() calls (though there would still be one additional exec() call).

But, really, argv[0] dispatching is plenty good enough.

This warning, however, is much too annoying.

Plus, argv[0] dispatching has worked for every other Unix and Unix-like OS of note for 4+ decades! What's so special about GNU in this regard? The answer is clearly: nothing.

What's the problem? Just make egrep and fgrep shell aliases in the default /etc/profile (if you're a distro maintainer).

Also, where's the extra fork?

    #!/bin/sh
    exec grep -E "$@"

the overhead isn't even an extra fork, as it uses exec

Good point. Though invoking $SHELL and parsing the script (as short as it is) can't be cheap either. Academically speaking of course; I'm not trying to justify the change on performance grounds.

On most context involving "using grep", shell scripts, interactive shell, etc, it's nothing.

81 Vs 128 system calls on my laptop (just printing --version)

    $ strace -fc grep -E --version 2>&1 | tail -1
    100.00    0.000000           0        81         3 total
    $ strace -fc egrep --version 2>&1 | tail -1
    100.00    0.000000           0       128         6 total

Measured with "time" 0m0.002s Vs 0m0.003s, always, testing 4 times each option.

Most situations where it could be being used, probably are surrounded by much bigger optimizations to work on.

true. using hyperfine on my machine shows that grep is about 2x faster than egrep (and yeah, still nothing, 1.5 vs. 3ms)

> I remember around ~10 years ago being told "you should never use `egrep` because it is slower than `grep -E`."

I remember around 30 years ago reading that egrep was faster (the UNIX versions were entirely separate implementations, egrep of course being newer) and have used it ever since.

exactly the same here.

although i'm using ripgrep more now, proving that you can teach an old dog new tricks.

I am not sure I agree with the logic. On an absolute instruction count scale grep will often dwarf the instructions necessary for the higher level logic captured in the shell script. Insert appeal to andahls law here. Writing in some more optimal language often requires considerably more work, and if the marginal benefit of reducing a few forks is minuscule compared to grep performance, why would you do this?

It depends on where your hot path is. If you’re writing the kind of performance critical code where the different between ‘egrep’ and ‘grep -E’ matters, then you shouldn’t be writing it in $SHELL to begin with. Not just because of the expense fork()ing, but because Bash is interpreted, because each exec requires loading new code into your L1 cache, and so on. It’s just never going to be as quick as something written in most other languages with a mature regex library.

I’m not saying this as a criticism against shell scripting (I’m an author of an alternative $SHELL and shell scripting language, so do have a love for shell scripting). I’m saying it as a criticism against people who try to optimise against the wrong things.

I guess in a way we are “violently agreeing” (as some say) because your point about the engineering effort mattering more than micro-optimisations is an example of the right kind of optimisations. Which is what I’m also, albeit badly, trying to describe.

> "if concerned with performance why are you running code as a shell script"

Rewrite grep in python before running - got it. :P

That's not what I suggested. I was saying if you're writing performance sensitive code with a hot loop calling `egrep` then a smarter approach might be to use a language better tuned for performance which supports regex libraries.

Shell scripts have their place and can out perform most languages if you're writing simple logic that can run across large datasets in parallel, such as

  cat very-large-file | grep "do not want" | sed -r 's/foo/bar/' > very-large-file-refactored

(please excuse the useless use of `cat`, it's there to illustrate the direction of data flow)

But in those types of scenarios the cost of $SHELL interpretation and fork() is massively outweighed by the savings of stream processing.

So my point was: if you're writing a function which $SHELL interpretation and/or fork() create enough of a performance impact where you're looking to optimize how you exec `grep -E`, then maybe it's time to investigate whether $SHELL is the right language to write your function in.

> please excuse the useless use of `cat`, it's there to illustrate the direction of data flow

Note that you can write

    < very-large-file

Instead of

    cat very-large-file |

To avoid a useless use of cat and keep the direction. Not that I care very much though. Using cat is less surprising to most people.

> Using cat is less surprising to most people.

Exactly :)

STDIN redirection is a neat trick though so definitely worth highlighting. But it wouldn’t have helped with readability in my particular example.

Perhaps I should have added a "/s", in addition to my ":p", but I appreciate the explanation here nonetheless

Why wouldn't they just convert them to aliases instead of adding the warnings?

aliases aren't a global state. They're shell and profile specific (as there are plenty of instances where a profile isn't loaded).

With `egrep` defined as a shell script, any process (even callers who are not shells) will have the same behavior.

That doesn't mean the shell script is the best solution though. Personally I'd rather they be a symlink to `grep` and have grep check the name it's been invoked as, and if it's "egrep" (or "e" if you're just checking the first byte) then implicitly enable the -E flag. This is how FreeBSD works and it feels like a more elegant solution in my opinion.

Sure, makes sense- so then I guess I'd amend my question to that: I can't see why they'd add the warning instead of just redirecting the command

wow. Didn't realize they are shell scripts on linux.

on (an older) macos system, they are identical executables