Benchmarking shell pipelines and the Unix “tools” philosophy | Hacker News Reader

Benchmarking shell pipelines and the Unix “tools” philosophy

blog.plover.com

112 pointsweinzierl63 comments

Thanks for this!

Another nice thing about /usr/bin/time is the --verbose flag which gives:

  Command being timed: "ls"
  User time (seconds): 0.00
  System time (seconds): 0.00
  Percent of CPU this job got: 0%
  Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
  Average shared text size (kbytes): 0
  Average unshared data size (kbytes): 0
  Average stack size (kbytes): 0
  Average total size (kbytes): 0
  Maximum resident set size (kbytes): 1912
  Average resident set size (kbytes): 0
  Major (requiring I/O) page faults: 0
  Minor (reclaiming a frame) page faults: 112
  Voluntary context switches: 1
  Involuntary context switches: 1
  Swaps: 0
  File system inputs: 0
  File system outputs: 0
  Socket messages sent: 0
  Socket messages received: 0
  Signals delivered: 0
  Page size (bytes): 4096
  Exit status: 0

Wow this looks amazing! I didn’t know time could track all of those!

man getrusage

Can anyone comment why you can only use the verbose flag if you use the full path of time?

    time -v ls

does not work but

    /usr/bin/time -v ls

does? I don't have enough knowledge of either linux applications or bash to know whats happening to cause this.

Your shell has a built-in "time" command.

man bash:

    If the time reserved word precedes a pipeline, the elapsed as well
    as user and system time consumed by its execution are reported when
    the pipeline terminates.

man time:

     Some shells may provide a builtin time command which is similar
     or identical to this utility.  Consult the builtin(1) manual page.

This is very likely because without the full path your shell is using the `time` builtin function of your shell as opposed to using the binary.

The shell's builtin keyword for `time` is more limited in nature than the full `time` binary. This is true of a number of other common unix commands as well, e.g. `echo`. The manpage for your shell should describe the builtins functions.

The full path calls a GNU tool, the other calls your shell's builtin function.

You can also type `type time` to see that time is a builtin. `which` and `hash` may also be interesting.

You can also use:

  \time -v ls

to skip the builtin version.

I got excited when I saw the 'f' and 'count' commands, but they're just scripts he has on his system. Like doing grep 'plover' blah.log | cut -d ' ' -f 11 | sort | uniq -c | sort -n. Personally I'd prefer to use the ubiquitous commands that work everywhere than rely on having custom scripts on my system, but they are nice.

That's the whole point of shell scripting, to take a series of minimal programs and tie them together into something that does a more complex task. There's no reason to distrust a shell script simply because it is a script any more than there is to trust a binary simply because it's a binary.

Sure, but relying on custom shell scripts as unix primitives can be problematic if you find yourself frequently managing/troubleshooting systems that you don't own, and you don't want to (or aren't allowed to) put those handy scripts in place. Then when you're on any given system, you forget whether you can use "f", or if you have to fall back on awk.

I think it's less about not trusting custom scripts than it is about ensuring that your unix muscle memory doesn't atrophy.

> if you find yourself frequently managing/troubleshooting systems that you don't own

You start thinking about packaging.

Packaging what?

I would be mad at any admin that would dare to deploy his helper shell scripts like “addup” and “count” on a machine other than his laptop.

And if you meant he could just have these things in his home, then it defeats the purpose of the original comment: trouble shooting and administering machines forces you to often switch user, machine, etc.

If a company has recurring troubleshooting issues, eg. "we need to know which process is taking all the CPU/RAM/IO", why wouldn't they add a script when they provision their machines with puppet or whatever tool they use?

Then instead of having to remember how to check all these things, they just run "find_resource_hogs.sh" and voila. It also enables other people to troubleshoot without specific knowledge.

Of course you don't want to put anything in there just because it might save 10 seconds, but then https://xkcd.com/1205/

I agree 100% with you.

I spend a lot of time moving around different machines, processes, configuration files, logs, etc. And I stopped maybe 10 years ago to use anything that is not available on a base system.

I don’t use fancy shells, I don’t use aliases, I don’t write local shortcut scripts, etc.

I just use regular bash, combine base utilities in one liners, and live with it.

Maybe I loose 1s here and there when writing one liners compared to someone with a library of wrapper utilities. But that gives me an immense benefit: I am at home on any machine, any distribution, everywhere, without any configuration, with any user.

There's always something like Ansible (or even a lower-tech solution) to at least give you the basic toolset that helps you double or triple your performance.

> that helps you double or triple your performance.

Sure, in a quest of productivity, I should also use Vagrant and Packer to create docker images with a development environment so that I can run Serverless troubleshooting containers on all my machines.

These scripts will surely help me triple the performance of my bash one liners.

I’ve heard Eclipse has good shell completion and support for oh-my-zsh too.

That’s the theory but frankly the syntax is so cumbersome, irregular and needs so many googling for "easy" things like conditional, substring, etc. that I now use a real programming language if a script needs to be anything more than a list of commands without any logic (besides variables substitution).

> That’s the theory but frankly the syntax is so cumbersome, irregular and needs so many googling for "easy" things like conditional, substring, etc. that I now use a real programming language if a script needs to be anything more than a list of commands without any logic (besides variables substitution).

You are basically describing modern programming.

Script Language (or scripting) is a programming language.

And about the "real" programming language you can also trap yourself googling and installing yet another library (did you read the code?) and/or reimplementing existing tools from the unix programming environment.

You are overly pedantic on a detail point that doesn’t matter: yes shell scripting is technically a programming language, but my point is that it is a terrible one worth ditching for any non trivial task. Perl was created precisely more than 3 decades ago to address this problem. Nowadays there are alternatives such as Python, Powershell or even scripting wrapper for compiled languages (such as C#) that allow to do the same job very well, with less surprising behavior and that can be refactored later more easily.

I disagree with you wholeheartedly and without condition. Not only are you discounting how much time it takes to learn how to program effectively in a real 'glue' language you are high handed in ignoring the ubiquity, working archive and efficacy of a shell script. Not to mischaracterize but I find this type of attitude most frequently in 'lead' individuals with less than 10 years experience: typically 20's and 30's in age. I'm curious if this is your case?

Shell may be super useful to learn but that doesn't mean it's full of horrible footguns that greatly hamper your ability to determine if your code is doing the right thing or not once you get to a significant body of code: just take a look at https://mywiki.wooledge.org/BashPitfalls and compare with other languages that don't have some of the more egregious easy-to-make mistakes (e.g. anything to do with word splitting).

I agree, and as TFA mentions, shell scripting is an "incoherent mess of composable paraphernalia" where you have to memorize a “weird mishmash of trivia” -- but if you have memorized it, then you can do some neat stuff without a so-called "real" language installed (that you have to upgrade and have to worry about if you have a modern version installed)

Most people who use Unix directly build up some stuff in ~/bin (often a misnomer because it's shell scripts and not binaries, although mine is less of a misnomer than most because so much is in C rather than shell). The trick is to build them out of the standard portable components that exist everywhere. (This means, among other things, no #!/bin/bash.)

sed 's| no | not only |'

I disagree.

You are also right.

Now that I saw tingletech comment I got your point :)

/bin/bash won't usually ship with a BSDish OS because of the license, so it is not generally portable to use bash-isms. (HPUX, IRIX, SunOS, Solaris, etc. I don't reckon would have had bash either)

Not to mention once installed on a BSD it would most likely reside in /usr/bin/bash (OpenBSD for instance)

Or /usr/local/bin/bash (on FreeBSD).

You shouldn't be using `#!/bin/bash` but rather `#!/usr/bin/env bash` instead.

I know this but usually don't do it. Been burnt by it too. Reprogramming your brain is hard.

[edit] Would there be any negative consequences to having an automated process go through and change it? Maybe the size difference might cause some issues with some things doing black magic with data in the file.

> [...] but they're just scripts he has on his system. Personally I'd prefer to use the ubiquitous commands that work everywhere than rely on having custom scripts on my system [...]

Is okay for one to have their own tools.

  $ f() { printf "\$%s" "$1"; }
  $ echo a b c | awk '{ print $(f 2) }'

His system is not very different from mine or yours. He just chose to combine the tools in a specific way.

Instead of doing

    grep 'plover' blah.log | cut -d ' ' -f 11

I usually do

    awk '/plover/{print $11}' blah.log

Less typing, but probably not as easy to understand.

"What if Unix had less compositionality but I could use it with less memorized trivia? Would that be an improvement? I don't know."

The answer is "no" here, because the alternative doesn't exist. Could it be created? Maybe in theory, but I suspect that the amount of stuff that you'd need to memorize (or learn to look up) to use it effectively would be about the same for any system that allowed a similar variety of work to be accomplished. If you are willing to trade off functionality for simplicity, then sure, it can be done. You can get it today by just not using all these tools at all, I suppose.

That is what the author says in the next paragraph.

> I don't know. I rather suspect that there's no way to actually reach that hypothetical universe.

I'm already living in that universe, but with even more composability.

There would be less trivia to memorize if the command behaviors and options were more consistent. You may not be able to achieve that at the edges, where new commands and options are added, but you can always go back and clean things up.

For example, the cut(1) command is intended to do precisely what his f script does. But it's inconvenient because unlike many other commands it (1) doesn't obey $IFS and (2) the -d delimiter option only takes a single character. This could and should be remediated with a new, simple option.

I think the only thing preventing that change is that there's not enough interest in moving POSIX forward faster; certainly not like JavaScript.

Another problem are GNU tools. They have many great features but OMG are they a nightmare of inconsistency. BSD extensions tend to be much better thought through, perhaps because GNU tools tend to be lead by a single developer while BSD tools tend to be more team oriented.

So the way forward isn't to replace the organic evolution, it's to layer on processes that refine the proven extensions. And we already have some of those processes in place; we just need to imbue them with more authority, and that starts by not rolling our eyes at standardization and portability.

Authority is the problem...not standardization and portability. Everyone is willing and able to tell you the best way to do your work if you use their tools. Straitjacketing implementation in the name of order is a surefire way to dissuade people from using your tools.

'sort | uniq -c | sort -n' is an interesting pipeline. It will always work and does a great job with large cardinality data on low memory systems.

However, if you have the ram, or know the data set has a low cardinality (like, http status codes or filesnames instead of ip addresses) then something that works in memory will be much more efficient.

I threw 144,000,000 'hello' and 'world' into a file:

  justin@box:~$ ls -lh words
  -rw-r--r-- 1 justin justin 824M Jan  7 15:21 words
  justin@box:~$ wc -l words
  144000000 words


  justin@box:~$ time (sort <words|uniq  -c)
  72000000 hello
  72000000 world

  real 0m22.831s
  user 0m32.999s
  sys 0m4.675s

Compared to doing it in memory with awk:

  justin@box:~$ time awk '{words[$1]++} END {for (w in words) printf("%s %d\n", w, words[w])}' < words
  hello 72000000
  world 72000000

  real 0m10.639s
  user 0m9.736s
  sys 0m0.876s

so, half the time and 1/3 the cpu.

This is because in the first example you are invoking two programs. The first one sort the content of the file, the second count how many lines are equal.

While in the awk example it is creating a hash table with all words and incrementing by the key and then printing.

There is no sorting plus printing may be buffered.

Thanks for explaining my own comment to me.

Not explaining, trying to tell that you are comparing apples to oranges and making a conclusion based on that.

Also, you don't need to spawn a subshell nor feed sort via stdin in the first example :)

He's comparing apples to oranges and reaching the conclusion that... yes, apples and oranges are different things. He's quite aware of this, and even points out the tradeoff -- `sort | uniq -c` still works if your dataset doesn't fit into RAM.

Counting "words" vs sorting+counting "lines" are a complete different thing.

Heck, forget about RAM, the output of both programs don't even match.

That awk is pretty efficient and fast is no surprise ;)

All of your examples work in memory.

Not exactly. sort (at least GNU sort) will end up doing external merge sort on temporary files if you give it more data than you have memory. Which, if you give it 100GB of 5 different strings, ends up being a huge waste.

Not only GNU sort, but also postgresql, mysql and many more...

Please, "huge waste"? How do you sort something that does not fit in memory?

Are you being difficult on purpose?

I posted a comment on how 'sort | uniq -c | sort -n' is an interesting and very capable pipeline, but often misused and slower than other alternatives.

> you are comparing

Yes, I am comparing two methods of accomplishing the same thing. That is how comparing things works.

> Please, "huge waste"? How do you sort something that does not fit in memory?

Note how the full sentence included "if you give it 100GB of 5 different strings". If your input is 100GB of 5 different strings, then the hash table will easily fit in memory, and sorting the entire data set only to pass it to 'uniq -c' is indeed a 'huge waste'.

There are tons of large data sets that only have a small number of unique values in particular fields. protocols, ports, http status codes, hour of the day, etc. 'sort | uniq -c | sort -n' will work for all of them, but not nearly as efficient a hash table.

> Are you being difficult on purpose?

Programming is about paying the bare minimum attention to the details.

> [...] two methods of accomplishing the same thing [...]

Absolutelly not.

one prints:

  72000000 hello
  72000000 world

the other

  hello 72000000
  world 72000000

Now try both examples against a file with more than one column to understand what I'm talking about ;)

Even working in memory, there are different efficiencies for different methods.

Awk includes an asort() function which can sort an array, such that it would be possible to create a similar process entirely within awk to the sort | uniq -c pipeline:

    #!/usr/bin/gawk -f
    { x[NR] = $1 }
    END {
        rc = asort(x)
        j=0
        for(i in x) {
            if( x[i] "" == x[i-1] "" ) freq[j]++
     else {
         j++
         elem[j] = x[i]
         freq[j] = 1
     }
        }
        for(j in elem) {
            printf( "%6i %s\n", freq[j], elem[j])
        }
    }

As compares with a hash-based counter:

    #!/usr/bin/gawk -f
    { x[$1]++ }
    END { for(i in x ) printf( "%6i %s\n", x[i], i ) }

On a 2,000 value test dataset with 10 unique values:

sort | uniq -c takes 0.019s (8 runs averaged)

awk hash takes 0.023s (8 runs averaged)

awk-implemented sort + unique takes 0.33s (8 runs averaged)

In this case, sort | uniq is the fastest option. But the all-in-memory sort + separate tabulation of unique values in awk is notably slower (running in 143% of the time) than the also all-in-memory hash accumulator.

As I bump up the dataset size (20,000 records) that discrepency increases, roughly 0.052s sort|uniq, 0.065s hash, and 0.217s ask sort-unique.

TL;DR: test your assumptions, especially regarding performance.

Note: Data were generated with a simple bash loop:

    for i in {1..2000}; do echo $((RANDOM%10)); done  > data

With super small dataset sizes like that it'll fit in the L2 cache and can behave differently. I did test this though, and for 20,000 items generated with your loop:

sort | uniq -c takes .017s (fastest out of a few runs)

the awk command I used above takes .013s

A trivial implementation I have in go takes .08s

Additionally, using this 'protos' file which is 1,000,000 lines of tcp,udp,icmp:

  $ time (sort protos|uniq -c)
     5915 icmp
   332003 tcp
   662082 udp

  real 0m0.232s
  user 0m0.739s
  sys 0m0.100s

  $ # fixed to count 'lines' and not the first column, which makes it faster.
  $ time awk '{lines[$0]++} END {for (l in lines) printf("%s %d\n", l, lines[l])}'  < protos
  icmp 5915
  udp 662082
  tcp 332003

  real 0m0.194s
  user 0m0.190s
  sys 0m0.004s

  $ time ./c < protos
  662082 udp
  332003 tcp
  5915 icmp

  real 0m0.088s
  user 0m0.084s
  sys 0m0.005s

so yes, I do test my assumptions.

I was suggesting @crystaldev may not have.

And, more significantly, and as you've confirmed, not all in-memory processing is equivalent. There are faster and slower all-in-memory algorithms and implementations

Hrm... Maybe a bogounique implementation might be appropriate here....

(By analogy to bogosort: https://en.wikipedia.org/wiki/Bogosort)

         for i in $(seq 1 20); do
           (run once and emit the total CPU time)
         done |addup

"Here we don't actually care about the output (we never actually use $i) but it's a convenient way to get the for loop to run twenty times."

This is slower than not running seq and just using builtins.

         n=1;while true;do
         test $n -le 20||break;
           (run once and emit the total CPU time)
         n=$((n+1));
         done |addup

The author did say convenient, not fast.

If you don't want the inefficiencies of seq, bash has:

  for (( expr1 ; expr2 ; expr3 )) ; do list ; done

which is a lot more idiomatic than constructing a for loop out of a while loop.

You could also just use a bash brace expansion:

    for i in {1..20}
    do
      ...
    done

https://wiki.bash-hackers.org/syntax/expansion/brace

The author refers to "the shell" in his "benchmark" but if he is a Linux user then

   sh -c 'f 11 access.2020-01-* ... is dash

whereas

   f 11 access.2020-01* ... is bash

Needless to say, one of these shells is much faster than the other and only one contains the "time" builtin command.

https://wiki.ubuntu.com/DashAsBinSh

Then there is the use of perl and its system command in "count". As with seq, why is perl needed. No explanation. Why not just put the entire pipeline into a perl system command.

I haven't been able to open this site for days (Timeout error, thought it was down), until today that I decided to use tor

> The appearance of the TIME=… assignment at the start of the shell command disabled the shell's special builtin treatment of the keyword time, so it really did use /usr/bin/time. This computer stuff is amazingly complicated. I don't know how anyone gets anything done.

Indeed.

the ACTUAL benchmark should be:

1) start timer

2) start deciding which commands to pipeline together

3) run the commands

4) stop timer

a lot of times the decision is the long pole.

in this authors case it included:

5) try a couple more variants of steps 2 and 3

6) write a blog post

This is a good point about the second half of the article (compositonality), but the author started the article by saying this was a command he "sometimes runs", presumably indicating he has it saved somewhere.

My commands of this type are usually retrieved using control-r

Reply on news.ycombinator.com