Hacker News Reader

Comment by jandrese | Hacker News Reader

Awk works great in pipes on the commandline, unlike Python where you would be writing a multiline script to accomplish the same task.

My 99% use case is extracting particular fields from some input stream.

  awk '{print $1,$5}'

That's a tremendously common problem and awk is the best tool for the job.

I once started diving deeper into awk and there is an impressive amount you can do with it, but I hit a point pretty quickly where it would be cleaner to write the script in a more traditional language where my successor won't be cursing my name for writing something complex in an obscure language.

View on HN

I like awk very much, but cut has one big advantage over awk - you can easily output range of fields also open ended ones. Example:

  $ cat foo
  foo bar baz
  foo qux
  $ < foo cut -d' ' -f 2-
  bar baz
  qux

As far as I know you have to loop through the fields, which is (haha) awkward.

If you just want to extract fields the cut utility might be a better fit.

    cut -d $delim -f $field1,$field2,...

Yeah, I usually start with `cut` - it makes the easy cases very easy - then only reach for awk if it turns out I need more power. Thinking of recent cases from my own experience, that tends to be needs like:

- outputting my selected fields in an order other than their order in the input lines: you can't `cut -f2,1`, for example.

- use a multi-character or regex field separator: -F'<space><star><comma><space><star>' [edit: rewritten in words to dodge HN's formatting] is a real winner when dealing with hand-written CSV.

- use a record separator other than \n: thanks, Windows, for all those times I've written ORS="\r\n" :P - you can even use a regex record separator, which not even perl supports; I don't need that often, but it's a godsend when I do.

- select based on criteria other than "number of fields from the start": I recently needed the last field from lines of varying field-count, and awk '{print $NF}' was just the thing.

But if you don't need anything like that, cut is a lot more succinct - especially if you need to split on tabs only, which it does by default, while awk defaults to splitting on spaces too.

I first learned awk precisely to replace cut. Not being able to reorder fields make it an especially narrow tool. Is there anything from cut I'm missing out on?

> Is there anything from cut I'm missing out on?

It's a narrow tool. Other than the inherent advantage this brings in clarity/simplicity in a script, there's nothing else to it.

Scripts in which you have both cut and awk are not simpler or clearer than ones in which you only use awk.

Although I completely agree with this for any given "one-liner" or pipeline (even if it's multiple lines), I don't believe the mere use of awk in one place in the script means using cut elsewhere isn't simpler/clearer there.

Consider a script that originally had now awk but just a

  field=$(somecmd | cut -d: -f3)

but now has been modified to have awk elsewhere. Did the above just become less simple or clear? If so, is it worth changing it to the below for clarity? Does it matter if the awk predates the cut, instead?

  field=$(somecmd | awk -F: '{print $2}')

I say "no" to both, although I do recognize the argument for just using the same, consistent tool everywhere, instead.

I'll tend to do this in scripts, both for clarity and precision (i.e. when I know the specific, single delimiter).

However, for ad-hoc command-line usage, I still prefer awk, because (1) the default delimiter is white space, which means I don't have to worry about if what I'm looking at might contain tabs intermingled with spaces (or how many spaces) and (2) it makes it much easier to incrementally increase the complexity of my ad-hoc operation, such as if I decide I want to take a sum or average of one of those columns, after all.

It's often easy to start with tools such as cut, sed, grep, tr, etc., but as you start building a one-liner (or short script) you realise that what you've cobbled together could all be expressed in awk.

You can shell out if necessary:

    cmd = "whois " $url
    while ( cmd | getline ) {
        # stuff
    }

    close(cmd)

Also: coprocesses.

The -w option in cut isn't universal which makes it unreliable for the extremely common use case of fields separated by arbitrary whitespace.

I actually wrote a sed utility to collapse arbitrary whitespace before piping into cut before I learned about Awk. I never knew about -w before, but looks like neither of the systems I'm currently using for work support it.

>I actually wrote a sed utility to collapse arbitrary whitespace

Can be done with tr, without writing a utility (if by "collapse" you mean what I think you do):

This file t:

$ cat t

the quick brown fox jumped

            over


            the lazy dog

containing many combinations of spaces, tabs and newlines (whitespace) can be changed to this file t2:

$ cat t2

the

quick

brown

fox

jumped

over

the

lazy

dog

by this tr command:

tr -cs "[a-zA-Z]" "\012" < t > t2

That also makes the output more amenable to further processing, including common tasks like finding the frequencies of the words in the input, as mentioned in the "More shell, less egg" post mentioned in this post:

The Bentley-Knuth problem and solutions:

https://jugad2.blogspot.com/2012/07/the-bentley-knuth-proble...

Does that work with utf8 files? I hardly ever work with us ascii files anymore.

Don't know. It was a while back, and in Python 2 (if you mean the Python version I wrote). I didn't take any special steps to support Unicode, so likely not. Same for the shell version I wrote.

I should've been more explicit, but was thinking about: tr -cs "[a-zA-Z]" "\012" < t > t2

(which I somehow managed to read as an awk invocation).

Re: python - I believe if using things like \w, \d or \s you should be Unicode safe.

Come to think of it, I seem to recall gnu tools should also have Unicode aware pattern/character classes, eg:

https://www.gnu.org/software/gawk/manual/html_node/Bracket-E...

> For example, before the POSIX standard, you had to write /[A-Za-z0-9]/ to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them. With the POSIX character classes, you can write /[[:alnum:]]/ to match the alphabetic and numeric characters in your character set.

https://docs.python.org/3/library/re.html#re-syntax

[ed:

Apparently gnu tr is still out in the cold re:unicode;

https://www.gnu.org/software/coreutils/manual/html_node/tr-i...

> Currently tr fully supports only single-byte characters. Eventually it will support multibyte characters; when it does, the -C option will cause it to complement the set of characters, whereas -c will cause it to complement the set of values. This distinction will matter only when some values are not characters, and this is possible only in locales using multibyte encodings when the input contains encoding errors.

]

Got it. Good info, thanks.

Just realized you may have meant the tr command I used - whether it supports UTF-8. Don't know about that either.

It's also use awk in that manner 99% of the time.

Another common usage for awk in my case is to do floating operations (*, +, eq, gt, etc) in a shell script. It avoids installing bc, which is not always present by default.

I used it for other things, like for example comparing software versions in shell scripts.

The most advanced usage of awk in my case was to do some statistic analysis (average, confidence interval, standard deviation) of network simulations.

Just be aware there are some caveats when using awk to operate on large numbers. GNU awk has the -M option to handle precision arithmetic, but where you can't use `gawk`, `bc` may be safer afterall.

Reply on news.ycombinator.com