Hacker News Reader

Comment by 0xFFFE | Hacker News Reader

I use sed & awk all the time. It is an invaluable tool while debugging issues, extracting fields from log files etc. I am not dissing python or perl, I use python extensively as well. But while you are in the middle of an incident, hard to beat a quick oneliner, like this random example.

awk -F":" 'BEGIN{total=0}{if($3>240)total+=$3}END{print total}' /etc/passwd

View on HN

Interestingly, perl was designed to fill exactly that role, with a syntax that was at the time considered cleaner and more intuitive than awk's (and of course with a regex backend that was much more powerful). In this case, something like:

    perl -e 'while(<>) { $tot += (split)[2]; }; print "$tot\n"'

It's funny to see that in perl's decline, the stuff it did well is being forgotten and resurrected via tools it at one point had mostly replaced.

Which is a shame, because I doubt I'll be loading modules from CPAN for awk any time soon, and I routinely use modules from CPAN or that I myself have written in my perl one liners. In fact, I have a specialized project-perl.sh in my project's PATH that loads a few modules (Try::Tiny, Path::Tiny, Data::Dumper, etc), sets a local::lib, and loads the main project module, allowing access to most the functions I've written for the current project, and passes all the extra params through to the underlying perl command.

If my codebase wasn't already in perl I wouldn't necessarily be able to benefit from a large library of my own functions relevant to the problem, but at least I would still have CPAN to fall back on. I'm not sure awk will ever be able to compete with that.

Let me golf this. :)

  perl -F: -anE '$tot+=$F[2]}{say$tot'

This made me miss perl (but ya forgot $F[2]>240)

Ah, I was going by the snippet in the parent, not the original one in the grandparent. That here should do it:

  perl -F: -anE '$_=$F[2];$t+=$_if$_>240}{say$t'

I wanted to be clever by doing `$t+=$_*($_>240)`, but that's actually one byte larger. :(

Perl does have command-line options to do that :)

    perl -F: -lane '$total+=$F[2] if $F[2]>240; END{print $total}' /etc/passwd

Of course it does, the implicit looping and BEGIN/END blocks (I'll concede I had actually forgotten about the implicit splitting) were in fact deliberately designed to emulate awk. Nonetheless I don't think that really changes the point much. The core language syntax is simple enough that perl one-liners can be written without that and still get the point across.

>>> I use sed & awk all the time.

Me too. For example, number of unique IP requests in a log file in milliseconds ;)

$ awk '{ print $1 } ' caddy_log | sort | uniq | wc -l

But rarely compose my own from scratch. It's mostly copy paste. And store in admin bin for future use.

No need so many commands :)

    $ cat duplicates.txt
    abc  7   4
    food toy ****
    abc  7   4
    test toy 123
    good toy ****

    $ awk '!seen[$2]++' duplicates.txt
    abc  7   4
    food toy ****

    $ awk '!seen[$2]++{cnt++} END{print +cnt}' duplicates.txt
    2

On a large file with many duplicates, seen[x]++ can overflow, unless you're using GNU Awk with bignums (gawk -M).

that's a good point

I'll add a note, thanks :)

Dumb side note: I always end up using `awk '{print $1}'` instead of `cut -f1` because cut's field separation defaults are unwieldy in that it doesn't intelligently consider any whitespace to be a field separator, which is what I need 99% of the time.

sometimes (if there are not mixed spaces and tabs) tr -s ' ' < input | cut -f1 helps

If you just want to count unique IPs, you can get that in awk alone and avoid sort:

    awk '{ip[$1]=1}; END{ print length(ip) }'

Use the hash tables directly, assuming you've got the memory. That should be slightly faster as it avoids the sort.

nice one

also, {ip[$1]} is enough...

Thanks, I was trying to think of the fastest possible assignment.

There's `sort -u` which should be able to avoid some work.

No need to spawn another process(uniq), you can use sort -u

;-)

For me, the benefit of a trailing uniq instead of a sort -u is that I'm invariably building towards a "sort | uniq -c | sort -n" anyways.

The fragment

BEGIN{total=0}

can be skipped (at least in the awk's I've used, not sure if some more recent / strict awk differs), because awk initializes the variable total to 0.

I remember this because I do this all the time to quickly get the (non-recursive) sum of the sizes of the files in a directory - it's pretty much muscle memory from a while now:

ls -l | awk '{ s += $5 } END { print s/1024 " KB" }'

For recursive size, one can use ls -lR or the du command with various options according to need.

But that only works because in the final print, you have done arithmetic with s.

So in other words, we can take out the BEGIN block, but then we must remember to change print total to print total + 0.

Also, using uninitialized variables is basically a code golfing stupidity that will bite you in any halfway complicated program.

GNU Awk has a useful --lint argument which spots uses of uninitialized variables. If you make it habit to write code that way, if you then use --lint for finding a bug, you have to deal with false positives.

>Also, using uninitialized variables is basically a code golfing stupidity that will bite you in any halfway complicated program.

Nonsense. Not if you know what you are doing, and used it in a known way, which is what I did. The code I wrote works. I tested it on Linux before posting it. Also, such a usage (skipping the initializer) is mentioned (IIRC) in the classic Kernighan & Pike book "The Unix Programming Environment" (still a great resource, though not updated for modern Unix/Linux features), which is where I learned it from, years ago (and hence why I qualified my statement by saying it may not work in more strict or modern awk versions). Fine to talk about other variations but it does not mean that my variation is wrong.

Don't try to read my mind. My intention was not code golfing. Was just sharing some fun info. It's not a big deal to keep the initializer either, I'm quite aware of that.

You do not know 100% what you're doing. Evidence being, in the grandparent comment you wrote "[...] because awk initializes the variable total to 0" which isn't how awk works at all.

Your intention can be understood as the promotion of code golfing, as evidenced by these words:

The fragment BEGIN{total=0} can be skipped

by which you're clearly encouraging that other coder to make their code shorter by removing an initialization that works fine.

>Your intention can be understood as the promotion of code golfing, as evidenced by these words: The fragment BEGIN{total=0} can be skipped by which you're clearly encouraging that other coder to make their code shorter by removing an initialization that works fine.

Your statement (above) can be "understood" as not understanding my prior statement(s), including the one in which I said "Don't try to read my mind" (w.r.t. intention, because you cannot - it is mine (mind), not yours). If you cannot grok that after a second explanation, I have nothing further to say. Good day.

Although your true intent may be anyone's guess, true enough, the prima faciae intent of forum posting persona is a fairly straightforward function of the content via which it portrays itself.

>code golfing stupidity

Smile when you say that, stranger!

http://www.thisdayinquotes.com/2011/05/when-you-call-me-that...

  #!/bin/sh
  x=0;while true;do read a;
  test ${#a} -gt 0||exec echo $x
  a=${a#*:\*:};n=${a%%:*};
  test $n -le 240||x=$((x+n));
  done < /etc/passwd

  #!/bin/sh
  IFS=:;x=0;while true;do read a b c d;
  test ${#a} -gt 0||exec echo $x
  test $c -lt 240||x=$((x+c));
  done < /etc/passwd

you could simplify by taking advantage of default value which is `0` in numeric context

    awk -F: '$3>240{total+=$3} END{print +total}' /etc/passwd

Reply on news.ycombinator.com