Hacker News Reader

Comment by xyzwave | Hacker News Reader

Somewhat tangential, but the author calls out using `grep` in conjunction with `awk`. Anytime I find myself doing this, it usually turns out that I can just throw the pattern into AWK itself.

    grep 'foo' file.txt | awk '{ print $1 }'

Becomes:

    awk '/foo/ { print $1 }' file.txt

There may be times when `grep` is preferable, but this is ubiquitous enough that it's mentioned in the "Useless Use of Cat" [1] awards.

[1] https://porkmail.org/era/unix/award

View on HN

Both the ‘useless use of cat’ and this nitpick miss that a unix-style command line is very much like a bunch of functions feeding results into each other, with each doing its own thing. This allows one to chain different functions in the same general way: namely, if I want to limit the input at first for debugging, I can do `cat stuff | head -10 | rest-of-commands`, or `grep stuff | head -10 | awk`—and I won't need to keep dragging the `<stuff` input around between commands, or to move the regex from awk to grep and back again or put the limit into the neatly written awk command.

Agreed, this is one the primary principles of the Unix Philosophy [1].

I was curious how performance would be impacted, and for a large file `grep` may not be so useless.

    $ du -h /tmp/file.json  
    161M /tmp/file.json

    $ time awk -F: '/"id"/ { print $1 }' /tmp/file.json >/dev/null

    real 0m17.810s
    user 0m17.690s
    sys 0m0.083s

    $ time (grep '"id"' /tmp/file.json | awk -F: '{ print $1 }' >/dev/null)

    real 0m3.617s
    user 0m3.641s
    sys 0m0.037s

While pushing the filter into AWK is a bit easier on the eyes, it appears there is an incurred performance cost.

[1] http://www.catb.org/~esr/writings/taoup/html/ch01s06.html

Yeah grep will generally be faster than awk for finding patterns in files, since that's the only thing it does, and it can (and does) take advantage of those constraints to apply performance optimizations.

Personally I don't think either approach is "better". Sometimes it's easier to deal with some complex logic if it's all in a single awk program, rather than smeared across combinations of grep, awk, and other things. If that outweighs the perf drop in some specific situation, then I'll do it.

You might get an even better speedup by using `grep -F` or ripgrep.

Can you share the file.json? I'd like to understand this difference better.

Unfortunately I picked something from my day job with a bunch of proprietary data.

It was essentially a large list of pretty-printed objects with about 20 attributes each. A few of which had multi-paragraph string values.

This is one of those things that really irks me about the Unix shell.

You're totally right that commands like `cat *.txt | grep foo` look right and feel intuitive, but the problem is that it's not actually equivalent to `grep foo *.txt`. With a single input file, it's probably not an issue, but with multiple files the downstream command can't actually see what file it's reading from, so in this example grep can't report filenames with matches and you can't use flags like --include/--exclude based on filenames.

I keep wishing for a Unix-like shell and utilities where `cat *.txt | grep foo` actually works the same as `grep foo *.txt`. PowerShell has pulled this off to some extent, which is cool, but it doesn't quite feel right to me compared to Bash.

Reply on news.ycombinator.com