Hacker News Reader

Comment by freedomben | Hacker News Reader

The new namespaces look interesting, although I really don't think the lack of namespaces it what has hindered Awk from widespread use.

I am speculating of course, but as someone who evangelizes Awk the most common thing I hear from people as to why they don't use it, is that they just don't know the language all that well. For people that don't know much about Awk, it looks really complex and esoteric.

To address the Awk ignorance, I put together a talk for Linux Fest Northwest last year, and it was so popular that I gave it again (virtually due to COVID-19) this year. The conference was fully remote so the talk is on Youtube.

If you've ever wanted to learn Awk, this will take you from zero to proficient. There are exercises as well for practice:

* Presentation: https://youtu.be/43BNFcOdBlY

* Exercises:

- Source: https://github.com/FreedomBen/awk-hack-the-planet

- My solutions with explanation: https://youtu.be/4UGLsRYDfo8

View on HN

> although I really don't think the lack of namespaces it what has hindered Awk from widespread use

I agree (I side more with Kernighan than Robbins here). My take on what hindered it from larger-scale use: lack of a big standard library (Kernighan more or less says this) -- Python and Go's stdlibs really helped them for example, and some significant quirks like lack of proper local variables (you can get them with extra arguments, but it's weird).

Just skimmed your AWK-learning talk -- very good stuff. I'm relatively proficient, but keen to try your exercises anyway!

Also agreed. I want the built-in Python data structures like dictionaries without getting too much Python DNA.

Awk arrays are dictionaries, what are you missing?

Never knew it lol, so thanks for pointing it out. It still stands that other functionality is missing.

The problem I think is that most people only use awk occasionally and that makes the skills hard to retain due to lack of practice.

I agree. And since you can have php (or whatever) on the command line, you really have to ask yourself "why should I spend 45 minutes or more going through old scripts I wrote to re-learn how to do what I want to do?"

I love playing with awk. I think it's really cool. But there's a cost to it.

I love AWK with a passion. It's just perfectly fills the gap between grep, sed and 'I better fire up a text editor'. I guess 'perl -pie' used to sit in that niche, but I've never used it much.

Very eye opening presentation and good exercises in the second video!

I had no understanding of awk whatsoever and can now fully grok the following command, which returns the number of different cities in a structured file with a header line, addresses and the city in column 5:

awk 'NR != 1 { print $5 }' payroll.tsv | sort | uniq | wc -l

This is beautiful.

Almost. It doesn't handle multi word cities, eg "San Franciso", and it spawns extra processes. Try:

  awk -F "\t" 'NR != 1 { if (!cities[$5]++) n++ } ; END { print n }' payroll.tsv

It is a very nice presentation! I did however see a lot of screen tearing. It looks like a partial flash of a terminal window.

I see[1]:

    Title: ben@bensystem76
    Dockerfile
    ngingx-help.txt
    test.pem
    Desktop
    part2
    drew.txt
    ....

I is only periodically but seems to be the same terminal tearing in[1]. I saw some in the beginning but now at 22:20 I am so annoyed that I come here to complain! ;-)

I checked both 720p60 and 1080p60 - I just pause at 22:20 - but plenty before that as well.

Maybe consider re-encode/re-upload if possible. If not: Epileptics beware!

[1] https://i.postimg.cc/6Q2RvbCn/AWK-screen-tearing-in-Youtube....

Aaaaand 1 minute later in the video you acknowledge the problem yourself.

Nothing to see here. Move along...

Haha yeah, so sorry. My OCD flares when I see that :why:

The reason AWK is not widespread is that it's only great for one case: working on a stream of lines. It's a pretty narrow niche.

Sure you can code anything with it, but there is always another language that will be better at this other thing, and still decent at what awk does. Awk looks more like a DSL than a generalist language.

So why learn awk to save you a few minutes for the rare cases you do need it? Just learn Python/Ruby/etc, it's a better investment, and it has a better crossplateform story.

> It's a pretty narrow niche.

If you're working on any kind of Unix, it's a niche that you encounter almost constantly.

> Awk looks more like a DSL than a generalist language.

Yes, it is pretty much the ultimate text processing DSL. So if you need to process a text file a line at a time, in most cases AWK is the optimal solution.

> and it has a better crossplateform story.

I think AWK is available on almost every platform where you can install those languages?

The reason AWK is so awesome is that it's only great for one case: working on a stream of lines

Regarding AWK being cross-platform, it is part of the POSIX standard.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/a...

Is there any reason to prefer it over `perl -pie` ? I found out that at the base level, they are pretty comparable; and at advanced level, CPAN (or perl- packages in distro's repo) have saved me more than once.

(Some embedded systems often do not have perl, but if they are that limited, they might have a very stripped down version of awk too)

If you already know perl -pie well, then it may not be worth it to switch to Awk unless maybe you do a lot of embedded development.

Awk is part of posix so is pre-installed and available nearly everywhere, even on many embedded devices. Busybox for example includes an implementation of Awl. Even my super cheap home router had an awk interpreter on it. Perl is also decreasingly commonly installed. On the latest Fedora for example I had to install it explicitly. Awk is still installed by default.

Not really. For many years, perl even shipped with a tool ("a2p" [1]) which would automatically translate awk syntax to perl -- there's very little distance between the two, syntactically speaking.

[1]: now available as https://metacpan.org/pod/App::a2p

One likely unpopular answer is that learning to use perl-pie well is more likely to lead one to create large perl programs, vs awk. Many of us feel those programs would be better written in Python.

Thanks for posting this. I'll give it a watch.

I bought an awk book and read it. Didn't see the point of the language any more. I have regexps if I need them, python if I need more. I rarely do much textual data munging anyway.

The exercises look bizarre - "Questions to answer about our payroll data using awk to analyze" - doesn't make sense. That's what DBs are for, or similar tools. It's just an exercise but still.

Final coffin nail is that having awk on your CV is useless, because everyone wants full-stack cassandra, spark and hadoop, and other stupid things. Awk won't help get you hired (try to find it in https://news.ycombinator.com/item?id=23042618)

Yes, the only time I come across Awk now is as part of some shell pipeline. It just seems to have been crowded out of lots of places. I still think it’s cool though.

Often old *nix tools are disparaged here by professional software developers. But not everyone is working in software development professionally, and not everyone needs sexy qualifications in other tools.

As an ordinary person whose day job is in the humanities, learning awk, sed, Emacs Lisp and other such tools has allowed me to quickly solve tasks that would have taken me ages to do manually. These tools have docs that are very accessible to the ordinary computer user (after all, they were conceived as something that clerical staff could be trained in). I am happy these tools are still around and I think they should be more widely known.

> I rarely do much textual data munging anyway.

Well, since that is pretty much AWK's entire reason for being, of course you don't have a use for it.

> That's what DBs are for, or similar tools.

But if you have a text file with a lot of semi-structured data and just need to answer a few questions, using AWK will almost always be faster than designing a schema and importing into a database first, before answering the question.

Oh, and want to know a really good tool for getting data from a text file into a database? AWK!

> Awk won't help get you hired

AWK just lets you accomplish many tasks quickly and without fuss that can create value for your company, and those are things that can go on your resume. At some point in your career, pointing to specific things you have accomplished becomes more important than the keywords on your resume.

> using AWK will almost always be faster than designing a schema and importing into a database first

I have never had to do that with unstructured data - ah wait, I lie! I pulled data off the covid wiki page, a quick emacs regexping, a bit of macrology, pasted it into a spreadsheet (libreoffice recognises it's tab or whatever separated) and bingo, you're off.

> Oh, and want to know a really good tool for getting data from a text file into a database? AWK!

For smallish amounts, I use emacs regexps. For data too large for emacs, I'd probably use python.

> and without fuss that can create value for your company, and those are things that can go on your resume.

That's not how it works any more. They look for skills. They should not, they should do as you say, but seems not any more.

> specific things you have accomplished becomes more important than the keywords on your resume

again, no. Believe me, I have a track record of delivering, but all my job agents do is match by skills. And that's what their clients want. It's depressing.

So I disagree but thanks anyway

For large data why do you use Python and not C++ or Haskell ? They can do anything that Python can do.

What's that you said, 'convenience and specialization', ah I see, that thought never occurred to me. May be that's why I reach for AWK over Python.

Awk, particularly mawk, is several times faster than python for a lot of file ripping tasks. If I had several billion lines of data to reformat or summarize, I'd almost always reach for awk before python.

> a quick emacs regexping, a bit of macrology, pasted it into a spreadsheet (libreoffice recognises it's tab or whatever separated) and bingo, you're off.

It's very likely that, if you knew AWK well, you could have completed this task faster.

> Believe me, I have a track record of delivering, but all my job agents do is match by skills. And that's what their clients want.

It's hard to find the few companies that actually prioritize results, but it's worth it.

Reply on news.ycombinator.com