Awk in 20 Minutes (2015)

In my job in VFX we are dealing with sequences of files constantly, and sometimes being able to do things to parts of the frame range is extremely helpful, and awk is great for that!

Let's say U wanted to delete exr files named like filename.1055.exr where the frame number is not between 1001 and 1100, that could look like this:

ls -1 *.exr | awk -F "." '{if($2 < 1001 || $2 > 1100) print "rm -v "$0}'

This actually just prints the commands out which is nice to check that it looks reasonable, then one can redirect that back to the shell by adding "|sh" to the end to actually run them!

The -F "." bit sets the field separator to dot, so that $2 is the frame-number part of the filename.

This is just the tip of the iceberg! Once you're over the initial learning-hump, there's so much U can do with awk! Great stuff!

Previous discussion: https://news.ycombinator.com/item?id=23048054 (762 points | May 2, 2020 | 126 comments)

See also:

* https://backreference.org/2010/02/10/idiomatic-awk/

* https://earthly.dev/blog/awk-examples/

* https://learnbyexample.github.io/learn_gnuawk/ (my ebook)

I work in HPC and mostly on the shell/console. Without a doubt, the combination of Bash (or another shell) and AWK is truly amazing. Being able to quickly generate statistics, filter out unnecessary information, generate pipelines, etc., is unmatched, and 99% of the time only requires a pipe redirect. Not to sound trendy, but using AWK really is the proverbial "if you know, you know".

One of my favorite use cases is based upon grep with extended regular expressions; there's always a need to search for strings while needing to exclude others, think of a basic example as "grep -E 'this|that' file |grep -Ev 'not(this|that)'". With AWK it's simple, "awk $0 ~ /(this|that)/ && $0 !~ /not(this|that)/' file". Or, if you're monitoring server load averages via a tool like sar, you can pick and choose which loads you want to monitor based upon a threshold, "uptime |awk '$10 ~ /[0-9]{3,}\.[0-9]{1,}/ || $12 ~ /[0-9]{3,}\.[0-9]{1,}/'". This will print matches if the 1 minute or 15 minute load averages are over 100.

Just because it's an "old" language doesn't mean it's obsolete or useless!

This is a great resource! I’ve used awk for only the most basic field wrangling. This makes me more confident in doing fancier things that I’d otherwise use grep for.

I learned from the article that one can assign to the dollar vars like:

$0="bye bye input line"

It seems like this could be a bit of a footgun!? I'm sure it must be helpful in some cases though! Has anyone used it? Could you share the use-case to help me understand this weird superpower?

Is there any tutorial like this for 'sed'?

Weirdly, I recently switched to Perl as a replacement for awk and sed, enjoying actual Perl regexps and very similar syntax for 1 liners.

I’ve had to deal with sed not accepting the same flags depending on systems and weird awk version too. At least it’s more consistent with Perl (which also happens to be installed by default).

Am I the only one?

The Awk Programming Language[0], while obviously much longer than the linked guide, is such a joy to read. One of the more interesting examples is the creation of a simple virtual machine in AWK.

0 - https://awk.dev/

I really like awk, I mean, it is not a perfect language, but it fits this really nice spot for a small language with enough features for useful everyday one off tasks. If I had to teach my mother one language awk is a strong contender.

I mean Look at the manual http://man.openbsd.org/awk it is like 5 pages. Admittedly, I grew up on obsd awk so that is my point of reference, gnu awk is considerably more complex. But I like how I can easily keep the whole language in my head.

Pretty clear and straightforward explanation for awk! That's a good article.