Hacker News Reader

Comment by yegle | Hacker News Reader

I was wondering if there's a CLI that can replace sort | uniq -c | sort -nr. I feel like this is a very commonly used pattern.

View on HN

Obviously you could create a shell alias or write a new utility for that -- but if you're wondering why there isn't a standard UNIX utility for doing that: All the standard text processing utilities were designed in the context of processing files which didn't fit into memory -- with the exception of sort (which uses temporary files) they're all streaming utilities which keep at most a handful of lines in memory at once.

The "normal" way of doing "count how many times lines are present and list them in decreasing order" is to use a hash table which keeps all the (unique) lines in memory -- so that utility simply wasn't possible when all the standard tools were invented.

Roll your own:

  ~/bin/freq
  ==========
  #! /bin/bash

  sort "$@" | uniq -c | sort -nr

Once you get into the habit of capturing your habits as scripts that snap together like German nouns, you'll find the command line a wondrous place.

Someone else mentioned using a hash table to accumulate counts; that approach would give something like

  ~/bin/freq
  ==========
  #! /bin/bash

  awk '{acc[$0]++} END {for (item in acc) {print acc[item], item}}' | sort -nr

To improve performance, consider using `export LC_ALL=C` (credit: https://benhoyt.com/writings/count-words/). But note that it sorts uppercase and lowercase separately, and doesn't handle Unicode well.

i've found data_hacks[^0] to be a useful substitute for interactive use. e.g.

  $ for x in {0..999}
  > do echo $((RANDOM%5))
  > done | bar_chart.py -np
  # each * represents a count of 8. total 1000
  0 [   193] ************************ (19.30%)
  1 [   200] ************************* (20.00%)
  2 [   209] ************************** (20.90%)
  3 [   193] ************************ (19.30%)
  4 [   205] ************************* (20.50%)
  $

[^0]: https://github.com/bitly/data_hacks

Maybe I'm missing something, but what's stopping you from creating a shell alias for exactly that pipeline?

The point is that it's slower than an equivalent do-it-all - sort | uniq is already slow for interactive use if you have a couple thousand input lines (with our computers it should absolutely never take more than a few milliseconds)

Who said anything about perf? @yegle said it was a common pattern and asked for alternatives.

> it should absolutely never take more than a few milliseconds

Why? It surely depends on your use case (highest possible perf is different for many dupes vs few dupes, for example). And a few milliseconds is lower than a human's ability to respond, so doesn't matter for interactive use, it can only matter in batch scenarios.

sort | uniq | sort -nr takes ~300ms on a file of 10k lines of random data that is 0.5MB on my 10 year old mac. It's ~60ms on 2k lines. It's faster on a newer computer, and unless you're doing this a thousand times in a row, there's next to no difference between 3 ms or 300ms to me in an interactive shell...

BTW, what is the alternative? Are you assuming I write something high performance, spend my time figuring it out? Or is there a higher performance sort|uniq already available?

> sort | uniq | sort -nr takes ~300ms on a file of 10k lines of random data that is 0.5MB on my 10 year old mac. It's ~60ms on 2k lines.

it's absolutely intolerable

> It's faster on a newer computer, and unless you're doing this a thousand times in a row, there's next to no difference between 3 ms or 300ms to me in an interactive shell...

having to wait 300ms after an interaction makes my hands shake. If the result is not on screen by the time the enter key has gone back up it's plain bad.

> it's absolutely intolerable

Huh. Okay. Maybe you could answer the actual question: why? That seems like a rather extreme position to take.

300ms was for 10k lines, not your example of 2k lines. 2k lines does come back within 2 repaint frames on my 10 year old mac, which is about as fast as most video game responses to controller input.

PS and I'm still wondering what your proposed alternative is...

> Huh. Okay. Maybe you could answer the actual question: why? That seems like a rather extreme position to take.

it makes me feel physically sick and my hands start shaking when a computer does not answer immediately

> 2k lines does come back within 2 repaint frames on my 10 year old mac, which is about as fast as most video game responses to controller input.

which repaint frames ? e.g. here on my 240hz monitor that would be 8.3 ms which i guess is tolerable for visual input. on a 60hz, that's 33ms which is noticeable when everything else around it is fast

> PS and I'm still wondering what your proposed alternative is...

? it's in the original post - having an utility which does such common piping use cases without having to go through three processes - here doing a time sort | uniq | sort -nr of a 100mb file gives me 2/3 seconds for the first sort, 1s for the uniq, and again a couple seconds for the second sort ; the uniq can stream but the sorts can't, and a single command would likely be able to optimize this with a bit of algorithmic thinking

> my hands start shaking when a computer does not answer immediately

I’m sorry to hear that. It sounds like it could be an anxiety issue somewhat unrelated to what we are talking about? Does it help to stick to very small files so you never have to wait? Does it help when you get progress updates in the interface?

I still don’t understand what you’re suggesting because the idea that you want to give arbitrarily sized inputs to a sort and have it come back instantly is unrealistic. You can only improve the overheads and bandwidth, you can never avoid waiting if you give larger inputs. No matter what solution you suggest, I can give you a file you’ll have to wait for. Speeding up the unix pipeline doesn’t appear to solve the problem you’re describing, that’s why I’m asking for more detail.

> it’s in the original post

What is? I’m not sure what you’re referring to. You only mean the imaginary idea to combine 3 processes into one? @yegle only appeared to ask for a shortcut for a common workflow, and not a more performant alternative. I’m asking you what your preferred practical implementation is. What can you do today to speed up this workflow? And how does that solve the problem of having to wait?

> the uniq can stream but the sorts can’t

Right, this is true regardless of what tool you use. You can eliminate the overheads of starting up 3 processes, but you can’t avoid the second sort, if you ask for a second sort.

Might be worth observing that the overheads of 3 processes get relatively smaller the larger the inputs are. If you’re sorting a gigabyte or terabyte of text, using a single process will be barely better, if at all.

Might also be worth looking at cold cache vs warm cache startup times for 3 processes. Does your 2/3 second change the second time?

PS GNU sort has options that can improve its bandwidth. Maybe try these? https://www.reddit.com/r/commandline/comments/a7hq5n/psa_imp...

This is another reason to avoid writing your own tool, since sort may in reality have a lot of functionality that is not easy to duplicate, and may have ways to gain performance that is hard to beat.

Reply on news.ycombinator.com