Awk is the coolest tool you don't know | Hacker News Reader

Awk is the coolest tool you don't know

portal.drewdevault.com

66 pointsTomte20 comments

While I agree that Awk is cool, I suspect most *nix folks over a certain age already know it well. I use it regularly to munge input data.

Its limitations force you to solve problems in a creative way, and it can be a good programming exercise.

On the other hand, it can't handle UTF-8, which is annoying. (Yes, I know about Gawk).

While "most *nix folks over a certain age already know it well", IMO it would be a shame for it to limited to them. I regularly encounter younger developers unfamiliar with standard tools like sed, tr, cut, etc. that I use often to solve everyday personal tasks.

The world took a wrong path when *nix tools were no longer seen as appropriate things for ordinary people to learn. (Yes, in the 1980s clerical staff in some companies were taught awk or even Emacs.) Productivity of the labour force would only rise. When my spouse moved to my country and didn’t yet know the local language, she worked for a multinational outsourcing firm. The job turned out to be little more than glorified data entry and it could have been easily automated with *nix tools piped. Hiring educated people to sit 8 hours a day and do everything manually was a waste of money and human potential.

> The world took a wrong path when nix tools were no longer seen as appropriate things for ordinary people to learn.

Traditional nix tools come with a lot of UX baggage. A full programming language is likely better for some tasks, though *nix tools still have their place.

I've been using awk for about 20 years, but I've only scratched the surface -- "fetch a few columns from this file" sort of stuff. A colleague recently shared with me a significantly complex program that he'd written in awk, and it was certainly impressive... but walking through it with him didn't exactly convince me (a huge esolang nerd) that I should spend my working hours writing such monstrosities.

In The Awk Programming Language (the original book), the authors create a fully fledged database in Awk. I agree, it's impressive, but probably not the ideal use case.

Your experience sounds similar to mine. Generally, my scripts are ad hoc and use only a handful of features at a time. They only need to be complex enough to get my data to the next stage in the pipeline, where another tool would probably work better.

And then you moved to use Perl...

Read the book while I was quarantine, still relevant today and very well written!

Edit: fixed link

https://ia902309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...

I get 404 on the above link, but this one works for me: https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...

awk is quite handy. I often use it in place of other tools like sed and cut depending on what I need. Usually I use it for quick oneliners, and on occasion bigger things. But typically more than few lines I end up reaching for a shell script or Python/Perl.

Minor simplification on the line numbering example, there's a builtin variable for counting records (NR or FNR) so no need to introduce your own counter:

   awk '{print NR "\t" $0}' < somefile

There are other builtins that are quite handy as well. The only thing that bugs me at times is when I want to print a range of records like column 5 to end, where the total number of columns varies. I wish there was a simpler way to do that like a column range selector. This is where I sometimes use 'cut' since its so simple to do column ranges (unless the column separators have consistent whitespace; then cut doesn't work).

I keep thinking "maybe I should have learned awk"... but I can do all of these things but the line numbering in sed (and there is a standard unix tool for line numbering called nl); it also feels like I might should have learned perl if I wanted this kind of power (but instead I made the "mistake" of getting very very good at sed).

As someone who speaks sed I thereby found this part super confusing:

> In this manner I used it as a kind of modified sed which only operates on certain lines.

But... sed already does that, with the same syntax even? (Note that I don't have BSD sed handy... I would normally use \0, but that & works fine in gsed.)

> awk '/^\/\// { gsub(/\[[a-zA-Z:]+\]/, "[&]") } { print $0 }'

> sed -e '/^\/\// { s/\[[a-zA-Z:]\+\]/[&]/g; }'

If you want to read the article how the author intended* (and not through an http portal) the gemini link is gemini://drewdevault.com/2021/05/03/awk-is-the-coolest-tool-you-dont-know.gmi

*https://drewdevault.com/2020/11/01/What-is-Gemini-anyway.htm...

Awk is nice when I need something more complex than just a straight up 'grep' (And has support for both basic and Perl regex) but I really think that sed is the best solution for almost all of the examples in just because of its readability vs standard regex and its power with streams.

Which flavor of awk supports perl regex? Gawk certainly doesn't

Oops sorry I was referencing grep not awk

awk is cool, and I use it from time-to-time but it is quite basic. I tend to prefer using Ruby (or Perl) instead.

For example I recently needed to check for log output where two columns differed. For a simple case like this awk would have probably done the job too but Ruby and Perl just have so many functions built in.

    ... | ruby -ne '$_.chomp =~ /^added (.*) ipfs\/([^\/]*)$/ and $1 != $2 and print'

Perl supports similar `-n` and `-e` flags. It even makes it a bit easy to do the regex filter and match (The argument is implicit IIUC) but I tend to prefer Ruby because Perl has too many sigils for a casual user to remember.

I find that for most of these use cases I'd rather just make a Vim macro. I already know Vim, so I don't have to learn and remember a new language.

For cases where it needs to be automated, Awk is probably the best tool. I don't know it though, and cases are few enough and far between so I've always just used Python.

I have seen awk be used for really cool stuff but I usually find myself just using sed and cut anyway. For example I would probably write the git ls-tree part as this:

     git ls-tree -r HEAD | cut -f2 | xargs -n1 sed -n "/^\/\//p;1q" | less

I think if a reader is not very familiar with awk they probably wouldn't be familiar enough with regular expressions to understand the magic of what the author is accomplishing. I could see this being insanely helpful if the regexes were explained a bit.

Someone used awk to make a generative synthesiser http://kmkeen.com/awk-music/

* Replace all `_` with `::` => sed 's/_/::/g'

* Append line number: `cat -n FILE`

Reply on news.ycombinator.com