Hacker News Reader

Comment by nrclark | Hacker News Reader

I'm always happy when I see posts that promote AWK. It's a very underappreciated tool in my opinion. I was a Linux user for 20 years before I got familiar with it. AWK is super powerful for text processing, and I like that it's included in Busybox for use on the embedded systems that I design.

For any complex text processing, it's way better and more robust than having a super long pipeline of a bunch of sed/grep.

Most recently, I used awk in a script that parses /proc/mount to grab the mountpoint of a partition, or print something different if the partition isn't mounted. Doable with a bunch of sed/grep and some shell logic? Definitely. But easier and cleaner in AWK, and equally easy to inline in a shell-script.

View on HN

I've never bothered to learn much AWK, but that's mostly because Perl is my bread and butter language and has been for 20 years, and focusing on knowledge of that seemed a better investment (especially since with a few judicious flags, Perl is a passable AWK replacement even for very small one liners).

That said, if you just want to supplement your knowledge of other shell tools and pull out something that can do some obvious text munging, AWK has always looked attractive for the task to me.

The problem is that awk is in POSIX, and perl is not.

There are two common sources of awk for Windows, for example, that drop one exe to provide the interpreter:

http://unxutils.sourceforge.net/

https://frippery.org/busybox/

Perl simply wasn't designed to do that.

But perl is available by default in almost every free *nix, and for most people, Windows isn't a requirement

Usually not a requirement but it's a "nice to have" sometimes.

In HP-UX 10.20, there was a single perl4 binary that dates from 1993 in /usr/contrib/bin (I checked our K380).

Had Perl held to a single binary, it could be inside busybox.

Perl cannot be inside busybox, and that hasn't been possible since 1993.

https://sourceforge.net/projects/ezwinports/files/

Has a recent version of GAWK (Gnu AWK).

I think if you install git for Windows, you get AWK as well.

> I think if you install git for Windows, you get AWK as well.

Yes, but you also get perl there too.

sorry, why this is different to install perl? they look like still need download. I s this because they were contained in larger tool box so more easier get awk by other reasons? p.s., thanks, I don't know I had have awk in git bash...

This comment reminds me of the joke about the sewage worker, who is chatting about his job and says 'to you it maybe just shit, but to me its my bread and butter'.

Internet always seems the similar, pipes|garbage in| and munging on urban dictionary.....

I do a lot of work with structured data--json, yaml, etc. For me, this is how I feel about jq. One of my favorite use-cases is querying Kubernetes resources. E.g., `kubectl get secret <secret-name> -o json | jq -r '.data | map_values(@base64d)'` (fetch a secret and decode all of its values).

Came here to say this. Glad to see /bin getting respect.

To anyone processing huge quantities of text and text files, someone very likely had the same problem you faced back in the 1980's and there's a Unix/GNU tool for it already.

I was introduced to *nix from processing very large text files that text editors I was familiar with choked and died. Someone showed me sed/awk/grep, and it took seconds to process when other GUI editors couldn't open the file. Never looked back.

For some purposes, awk+xargs can replace hours of work to write a tool to automate some process. It's my go-to for ops work that I don't expect to live very long and just needs to _happen_.

Also, happy 1337 karma day :).

> awk+xargs can replace hours of work

Including machine hours of work.

Wasn't there a famous story of replacing a Hadoop cluster with an awk script (which was a couple orders of magnitude faster)?

Oh yes, there was: https://news.ycombinator.com/item?id=17135841

In fairness it's xargs that is providing the command parallelization, not awk, but I agree both combined are a good match.

If one considers the idea of map reduce to be taking a set of data and ending up with a subset that is relevant, I've used tons of simple things to do that, and never Hadoop.

I think parsing logs to find pain areas or potential exploit/exfil is a map reduce job, for instance, and grep or awk can manage that just fine.

Not having to parse the output at all is even better. I really like the way Powershell can pass structured data like this.

I'm a huge Linux/Unix fan but sometimes a rethink really works out. I hope Linux will get something similar. I know Powershell is available for Linux but without an adapted userland there's not much benefit

I have used it for many thing such as extract bibtex entries and checking for missed definition of abbreviations, converting remind entry to ICS format etc. Recent example is to print a calendar like cal but with calendar dates highlighted.

Yep awk is lovely and well worth the time to learn.

This is probably not important for embedded but doesn’t a pipeline of small scripts (which could be in awk) give you better threading support?

Xargs, GNU parallel or even make then scale that out really quickly.

Reply on news.ycombinator.com