Hacker News Reader

Comment by gbrown_ | Hacker News Reader

I certainly agree with your points, though the original task is a textbook example of where awk shines.

    awk 'BEGIN {FS=","} /cake/ {print $2, $3}' *.csv > cakes.csv

Unsurprisingly I disagree with the post's description of awk being an "advanced command".

View on HN

>I disagree with the post's description of awk being an "advanced command"

I guess there's no categorization of "advanced" vs "beginner" that will satisfy every audience but I consider awk an advanced tool. About 20 years ago, I wrote some AWK tips and cheatsheet back on USENET and today I would have to refer to that post to write basic awk commands.

The thing about awk is that it's a compact programming language with variables and conditionals and that's a step-change in complexity for many users.

> Unsurprisingly I disagree with the post's description of awk being an "advanced command".

I think it can be pretty advanced, for me awk is one of those tools where I still feel like I need to write a paragraph of comments to explain 1 line of code.

For example: https://github.com/nickjj/invoice/blob/75660dce5a29ceb4e47a6...

Keep in mind I don't really "know" awk. I cobbled that together from a few examples. It will convert times formatted like "2h 30m", "150m" or "2:30" into 2.50. There's a bunch of examples in the test file.

NOTE: I wrote that script 2.5 years ago and I know there's questionable patterns in other areas of the script that's not highlighted like using a bunch of separate echo calls instead of a heredoc.

Shell scripting is really fun and efficient. I use it all the time for a variety of things.

awk is Turing-complete, so if that is not enough to describe ”advanced” tool, then I don’t know what it is.

So is sed.

I’d classify basic search/replace functionality of sed as “basic” - you see such scripts frequently enough in shell scripts. Same for awk and “extract fields 1 and 3”. Anything beyond that quickly escalates into intermediate and advanced usages. Often you can find an SO post for lots of intermediate usages (where you can just include it as a link saying “I’m trying to do X”) but advanced usages that require knowledge of that are best avoided unless you are investing in sed/awk expertise in your team. Usually choosing something like Python, Ruby, Node or even Perl can provide better value because all of those languages can solve the same problem and solve other problems those tools can’t.

  awk -F, '/cake/ {print $2, $3}' *.csv > cakes.csv

does the same thing: -F sets the field separator.

awk is one of those underrated tools I always wanted to learn.

I still remember when I started working as a sysadmin at 19, the greybeard UNIX guy taught me how to vim, and he told me awk was as important as knowing vim, pointing to some huge AWK manual he had on the shelf, one of those with the animal in the cover.

This was 15 years ago, I know vim, but awk still eludes me.

I wrote a book on GNU awk one-liners with hundreds of examples and exercises [0]. Free to read online.

For a quick introduction, see [1] [2]

[0] https://github.com/learnbyexample/learn_gnuawk

[1] https://backreference.org/2010/02/10/idiomatic-awk/

[2] https://earthly.dev/blog/awk-examples/

Read The AWK Programming Language, a joy to read, one of the finest docs ever written, I reckon. You'll be up and running in minutes.

https://archive.org/details/pdfy-MgN0H1joIoDVoIC7

If you know C and regex, that's 85% to 95% of awk. The basic layout of an awk program is a sequence of regex/action pairs, where the regexes are between slashes, and the actions are between curly braces. (There's also the special regex values of BEGIN and END, executed prior to the first line of input and after all input is consumed, respectively). The action part has a syntax very much like C, with the special values $n (where n is an integer) representing field number n for a any input line. There are a bunch of special functions it defines that you can call, like substr(), or split(), and good old printf. Awk processes the input stream one line at a time, matching each line against the series of regexes, and if it matches a regex, it executes the action paired with that regex, with all the $n variables filled in with field values from that input line. And that's basically enough for 99% of what awk is used for on a day to day basis.

I wrote one script in awk and learned a lot:

- I understood where a lot of the stuff in Perl (and, to a lesser degree, Ruby) came from, especially Perl's implicit loop mode.

- I never wanted to write a program in awk again. Ruby (or whatever your preferred scripting language is) is better every time: awk is not much less work and Ruby generalizes more.

Awk for scripting is a bit like shell for scripting. It's easy to extend a script into something which is uncomfortably expensive to rewrite when you inevitably want to do something it's not well suited to. Awk still has uses, but it's in things like portable shell scripts which need more text manipulation power.

My thoughts exactly, though with “-F,” to quickly set the field separator.

Awk is advanced in the sense that it is a programming language by itself. Years ago I had to migrate an old version of an Informix database to a newer version. The old version did not have tools to export the database as DDL and DML statements. So I had to create a tool myself to do it. The system did not have perl installed so I had to use awk. It worked nicely enough.

I love awk. I've been using it for over 25 years.

But awk is usually not a good choice for processing CSV files.

Your program will fail if any of the fields include embedded commas (which is often the case).

Reply on news.ycombinator.com