Hacker News Reader

Comment by ufo | Hacker News Reader

The blog makes two main points: 1) adding new error messages causes compatibility problems; 2) some people are used to typing "egrep".

On the first point the author only gives hypothetical examples. I feel the argument might have been more compelling if we could see some concrete examples of things that break with GNU Grep 3.8.

As for the second point, I find it less convincing than the first one. If it's just the muscle memory then an alias would be an acceptable workaround. And I doubt that "everyone in the world" would want such an alias, as the author suggests.

View on HN

It seems pretty simple, piping bash commands into other bash commands and other text stream juggling is a pretty typical use of these commands and so changing what stream is output can change the behavior of consumers of the output of these functions.

I haven’t done anything with fgrep and egrep before but piping grep into another grep for more complex classes of text search is something i use a lot.

It's more than likely the warning will be printed to stderr, not out, so there will be no impact on the actual work done.

Automatically monitoring the stderr from cron jobs for unusual outputs is a prudent measure, and its plausible that this change will increase the burden of false positives (it certainly will not reduce it.)

But if you’re monitoring the output it usually means you are in a position to fix problems which means you can likely update the script in question to use the new warning-less invocation.

If I had been woken up in the middle of the night or had a vacation interrupted on account of this, I would not be entertaining warm and grateful thoughts toward whoever thought it was a good idea.

Wouldn't you test before upgrading packages in production? And usually you'd want to schedule any upgrades so that the next day or two has coverage from someone who can deal with any issues that arise.

What if a cron.monthly or cron.weekly script calls egrep? Congrats now you get a lot of noise from cron stderr emails in the distant future.

I have yet to work at a place that didn’t have systems running mission-critical shell scripts with little to no SDLC on boxes that got periodic “yum update -y”s. There seems to be a difference in oversight of software we write & “the operating system”.

Should we do better? Absolutely! Will this burn people if vendors don’t take care? Also absolutely!

I don't know if I have sympathy for this argument.

Your script ostensibly handles (at least logs) errors and warnings right? Do you exhaustively handle every single error and warning in a unique and different way or do you have a catchall "If non-0 return code then fail"? How does introducing new output to stderr affect that?

If you're being woken in the middle of the night over this then your testing infrastructure is crap.

This is something that should be caught before it gets to that point SPECIFICALLY so you aren't getting woken up in the middle of the night.

All the replies so far are missing the point: it is prudent to monitor for unusual events, including previously-unseen messages, over and above the explicit handling of specific errors. It is also prudent to not wait until morning to investigate.

Your infrastructure probably is crap, as very few people get to build it themselves from scratch. That does not mean one should cheerfully accept additional unnecessary or pedantic complications.

It would also be prudent to investigate each and every change to any of the software you use, in order to anticipate problems, but unnecessary and pedantic changes increase the burden there, as well.

hence why I wrote "actual work done".

your scripts will continue to produce the expected output. the side effects, otoh, will change indeed.

It's not unusual in shell scripts to combine stderr with stdout by using "2>&1" or similar.

It is, however, very unusual to do so and then try to parse the output. Aside from compilers, what other CLI tools make any guarantees wrt what they print to stderr?

Can confirm:

    $ egrep '.' < <(grep --version) > /dev/null
    egrep: warning: egrep is obsolescent; using grep -E

The simple fact that you have to wonder about that question is the failure.

Everything about this, even this comment I'm writing right now, is a waste of time.

With regard to the first point, the examples may be hypothetical, but they are also very plausible.

When a change has little or no objective benefit, I feel the burden of demonstrating that it is harmless falls on those making the change.

As has been pointed out elsewhere, this is free software and the maintainers are free to do whatever they like. That does not stop others having an opinion about it, especially when it is in the form of constructive criticism.

Sure, but it would still be nice to have at least one such example. Looking at the rest of the discussion thread here on HN as of now it's still only hypotheticals.

It would be even nicer to see convincing evidence that it is not going to be a problem.

You can’t prove a negative.

There is no largest prime.

Using that model, we can prove conclusively that, since a behavior has changed (a warning printed), it might cause problems. Therefore, we cannot prove that it cannot cause problems. What we would really like, though, is an actual problem shown to exist. Just like in mathematics; it’s one thing to prove that it’s impossible to prove something could not exist, but another thing entirely to show it existing.

You are overlooking something here: I never said anything about proof. I explicitly wrote 'convincing evidence' because proof is too demanding!

It's rather amusing how you have flipped from saying "you can't prove a negative" to an argument for the certainty of observable effects and the probability of consequences! (you wrote might cause problems, but everyone can see that's an unrealistic understatement of the implications of the argument you are using.)

The NASA managers prior to the Challenger crash thought that what they really wanted was something showing them an actual problem existed. Erring on the side of caution is generally prudent, even in relatively small matters.

> everyone can see that's an unrealistic understatement of the implications of the argument you are using

If nobody can show an actual existing problem, or even an example of reasonable code someone could have written which would be impacted by a the printed warning, then yes, I would think that I was charitable when I wrote “might cause problems”.

What does 'charitable' mean here? Generous towards what person or point of view? As you say you have conclusively proved that there is a non-zero probability of there being problems, the use of 'might' is already trying to persuade that this possibility is next to zero.

I thought I was being charitable when acknowledging that there might be a chance of a problem, when I in fact believe there not to be one.

If you were presented with an actual case, would you change your mind over whether introducing this warning is advisable?

Yes, of course. If the case is reasonably likely to occur in the real world and have real world impact, that is. And I would assume that the GNU grep developers would agree with me.

I think that is a very reasonable position to hold. It would only be the rejection of plausible cases, on the basis of no actual case having been uncovered, that I would take issue with. When assessing the downside of a proposal, there should not be much, if any, difference between how highly plausible and certain consequences are assessed.

If we could prove there would be no downside, then plausible problems could be ignored as merely hypothetical, and there would be no need to posit offsetting benefits. The question the GNU grep developers might want to consider is whether the supposed upside will have sufficient material consequences for their purposes.

Here's one example. This is in code my team inherited a long time ago, and there are many more like it.

        databases=`find /var/lib/mysql -type d | sed 's/\/var\/lib\/mysql\///g' | egrep -v 'mysql|test|performance|schema'`

That doesn't do anything with stderr, so it doesn't break.

It does output to STDERR an extra warning.

But that doesn't actually break your script, because backticks only capture stdout.

But it changes the behavior of the script in the UI. It can cause things like cron to send mail. It can cause other things wrapped around the script that are capturing both STDOUT and STDERR from the script to capture extra content. Any tool that's monitoring STDERR and expecting it to be empty may consider that an erroneous run, which may impact other scripted decisions. It's a breaking change in multiple circumstances, even if you don't consider extraneous warnings shown to a user manually running a script a breaking change.

Does that code look like something you'd log into a system and manually run on a regular basis? Does it maybe instead look like one layer of a legacy automation stack absorbed into other tools?

> On the first point the author only gives hypothetical examples

I have scripts everywhere, some of them 20 years old or more, that use fgrep. For years and years it was a "best practice" thing if you were checking for a fixed string (so that you didn't accidentally match on a "." or whatever by forgetting it was a regex).

You have two options: The first option is to simply replace "fgrep" with "grep -F" everywhere in all your scripts, which is correct but is more work than your other option, which is to add your own "fgrep" script somewhere in your path.

Any of these options seem reasonable to me.

OK, are you going to call up all my former employers to tell them to audit the scripts I wrote for them in the late 90's?

I really don't think people understand the impact here. It's not it's just a bunch of angry geriatric graybeards yelling at the modern world. It's that there is decades of uncounted, unrecognized, untraceable software written using these old conventions that are suddenly changing.

It's just a terrible idea. Linux cares about conforming to syscall interfaces for binaries compiled 20 years ago, but somehow you think it's OK to break scripts that have worked fine for 50 (fifty!) years?

Either a system is frozen and static, in which case it will not receive this version of GNU grep, and there is no problem. On the other hand, if a system receives updates, the system needs both minor and major changes all the time, to keep up with its ever-changing environment. This is the jungle in which we live. Linux syscalls are important to keep, since it’s hard to change a compiled binary. But it’s easy to change a shell script.

And don’t exaggerate. This won’t, in all likelihood, “break” your scripts.

> This won’t, in all likelihood, “break” your scripts.

Previously:

> The first option is to simply replace "fgrep" with "grep -F" everywhere in all your scripts, which is correct but is more work than your other option, which is to add your own "fgrep" script somewhere in your path.

    script.sh: line 100: fgrep: command not found

seems like evidence of a broken script to me. The fact that it can be fixed doesn't make it not broken.

We were discussing the warning printed by fgrep and egrep, not their removal. I was suggesting that you put a version of fgrep/egrep in your path which does not show the warning.

That wasn't my take. I thought we were discussing the deprecation itself. It's true that nothing is broken yet[1]. But it's clear that "broken" is where we're going, and I don't think you or the GNU maintainers have thought through the implications correctly.

[1] At least, nothing that isn't sensitive to junk on stderr -- bourne scripts are not always as tolerant as you'd like.

We were explicitly discussing GNU grep 3.8, which does not remove anything, only add warnings. And the remote possibility of breakage due to warnings is why I qualified my assertion with “in all likelihood”.

The shell happens to be a programming tool and not just an interactive command interpreter. There are whole books on how to write shell scripts portably across various Unix-y platforms. Many of the examples in those books will now throw warnings on systems using the GNU tools.

If a book on writing portable shell scripts across *nix platforms depends on commands not specified by POSIX, I don't think those books are doing their job very well.

POSIX.2 wasn't a standard until 1992. Perhaps it's the standard at fault for specifying the -E and -F flags rather than specifying the tools that existed in v7 in 1979. Authors and publishers didn't just pause for 13 years to wait and see.

Reply on news.ycombinator.com