Hacker News Reader

Comment by veltas | Hacker News Reader

The only really horrid thing about awk is the way you declare local variables in a function ... by just adding more parameters (and letting them be default-initialised on calls). Otherwise it can act like a relatively conventional scripting language, aware of associative arrays (ahead of its time).

View on HN

Back in Google New York, sometime after 2006, but prior to my team changing floors after the YouTube acquisition, my team sat about 2 cubicles over from Peter Weinberger, and a bit further from Brian Kernighan's office. My manager, John Sarapata was standing up and complaining how much he hated awk, and I ducked a bit and said "You know AWK stands for Aho, Kernighan (motion toward Brian's office), and Weinberger (motion toward Peter's desk), right?" They're both very nice people, so I don't think they would have been too offended, but it just felt odd to be boldly talking behind the backs of such well-recognized luminaries. Last I checked, John was very senior in Google NYC, so I guess he's a bit more careful to get full context before complaining loudly. Good times.

1. This is an incredible story haha. 2. I constantly talk about how much I hate python or other languages that I actually like. Sometimes the things you complain about most are the things you need most too

"There are two types of programming languages: those everyone complains about, and those nobody uses."

- Bjarne Stroustrup.

I agree with him. His book "The Design and Evolution of C++" was a fascinating read. His site has an overview of it.

A big problem with Awk is that it lacks garbage collection (including ref counting), and that puts major limitations on the language. You can't return an associative array from a function:

    $ awk 'function f() { a[1]=2; return a } BEGIN { f() }' </dev/null
    awk: cmd. line:1: fatal: attempt to use array `a' in a scalar context

You also can't have nested associative arrays, i.e. recursive or cyclic data structures are not allowed.

As far as I can tell, this is because a stack frame owns everything allocated within it, and when a function returns the whole stack frame is cleaned up unconditionally. You can pass arrays down but not up. It's very naive memory management (by modern standards; it's probably better than BASIC).

So I'd say it's clearly not expressive enough for general purpose programming. There are Lisps in awk but they do weird tricks with text as far as I remember.

Thank you. This comment alone is enough reason to never touch Awk if I can help it.

You definitely want to work with it if you can. I live much of my life in python, but there hasn't never been a week, and usually not a day, in the last 10+ years where I don't write a quick awk snippet to do something useful to data.

The advantage of awk over python is that it almost always takes < 60 seconds from the point at which you say, "I want to summarize, review, filters, report" on some columnar data to the point at which you have output in front of you.

I"m fast with python doing that - but it's usually a 3-4 minute set of steps.

The only exception being json, which awk is clumsy with. (jq is my goto tool for the simple queries, python for anything that requires more than a minute of thought.)

Also - it's almost always the case that I'm using awk in conjunction with a bunch of other tools, cut/sed/uniq/grep to get what I want.

> it's clearly not expressive enough for general purpose programming

The problem of persisting arrays usually leads to handling them in bash.

Now you have two problems.[1]

[1] If you paraphrase Jamie Zawinski in an explanation, you then have two things to explain.

Not sure what you mean because bash has the same problem -- you can't return an array from a function.

And I still think Shell, Awk, and Make Should be Combined. In that world you'd still have one problem :)

https://www.oilshell.org/blog/tags.html?tag=awk#awk

Oil has recursive data structures, although I'm still working out how to compose shell-like "procs" and functions which could return an associative array. We could just add them both naively but I think it would cause many shell programs to be a lot messier.

You don't have to in bash because you can just declare that your array is global inside your function. In fact, this is the default behavior.

Sure but you can also declare global arrays in Awk. The point is that Awk and bash are the same in several respects, so you don't really gain anything by switching to one or the other:

    - they don't have any garbage collection or ref counting
    - they don't have recursive data structures
    - you can't return any kind of array from a function

In bash you also can't pass an array to a function, but in Awk you can.

Many BASICs had GC to clean up strings -- even on rinky-dink 8-bit micros.

> There are Lisps in awk but they do weird tricks with text as far as I remember.

Not quite -- https://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/... was the first and used two global associative arrays, car[n] and cdr[n], where n is an integer which iirc had to have a particular tag modulo 4, as the type tag. (Much as you'd do it in a lower-level C implementation.) I used almost the same scheme in my later Lisp.

Yeah this is cool, I was thinking of the "normal" way of doing it where evaluating a Lisp function reinvokes the interpreter function. But there are obviously other ways to do it if you have global arrays, as awk does.

I have really warm feelings about AWK, actually! It’s such a simple language that you can learn pretty much all of it in an afternoon, but it is still expressive enough to be really useful as a text processing language. I have an AWK script that turns my CSV bank statements into Ledger transactions, and AWK is just perfect for that kind of thing.

Didn't mean to sound negative about it, I love awk too. It is definitely small enough to learn in a day (or two), and it fills a niche people don't realise they have.

LOL that's what I do too.

A bit of pdf2txt, awk and some shell scripting, and budgeting becomes very easy...

Can you share? I've been thinking of doing the same and would love to see some prior art.

be wary that the initial parsing is probably going to be bespoke for your bank account csv export format. my own bank doesn't even share consistency in these exported csvs between the checking, savings, and the credit card I have with them.

Awk more or less introduced associative arrays to modern programming.

SNOBOL4 was earlier than AWK with associative arrays by a little under ten years. However, from the standpoint of manipulating strings with regular expressions, AWK introduced the concept as far as I know.

String pattern matching in Snobol4 was still stronger than regex for a very long time, possibly still today. A language very ahead of its time.

Yep, I recommend Gimpel's old book Algorithms in SNOBOL4 for fun examples (from the days when such a title did not mean an undergrad data-structures-and-algorithms curriculum).

Agreed it is an excellent book. Even the Catspaw Spitbol manual is well written. Along with the Green Book. An often forgotten period of computing!

I did say "more or less" and "modern". Who's heard of SNOBOL4 now? :) My guess is that every other language that got AA's got them by way of influence from awk (or something else that got them from awk); awk itself may have gotten them from SNOBOL4 though.

But ok, we can at least say that awk is the oldest programming language to have associative arrays that's still in widespread use. (Yes, I'm hoping that someone will argue for SNOBOL4's current widespread use. :) )

> However, from the standpoint of manipulating strings with regular expressions, AWK introduced the concept as far as I know.

Two can play at this game! `awk` is really a successor of `sed`, which is all about manipulating strings with regular expressions. sed maybe isn't turing complete though (but someone's gonna prove me wrong here too) or at any rate not convenient to use in as general a way as awk.

> sed maybe isn't turing complete though (but someone's gonna prove me wrong here too)

Done! :) Since a turing machine has been written in sed, sed is turing complete. [1] [2] (both from Peter of Browserling).

  [1] https://catonmat.net/proof-that-sed-is-turing-complete
  [2] https://catonmat.net/ftp/sed/turing.txt

Thank you, kind friend.

> Christophe isn't the first person to realize that sed is almost a general purpose programming language. People have written tetris, sokoban and many other programs in sed.

Wow! But clearly, you wouldn't WANT to except for the fun of it.

I wouldn't want to :). It looks very expressive, much more than I had known just 3 days ago, but maybe it's a little loose around the edges for my taste.

Reply on news.ycombinator.com