Hacker News Reader

Comment by sciurus | Hacker News Reader

This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm.

https://en.wikipedia.org/wiki/Comm

  # show only items in both a and b
  comm -1 -2 a_list b_list

  # show only items unique to a
  comm -2 -3 a_list b_list

  # show only items unique to b
  comm -1 -3 a_list b_list

View on HN

I love how, even after using Unix-y systems for probably 10-odd years, I still occasionally stumble upon something like this that refers to a command that I've never even heard of, and when I check (on my Mac, no less), it's right there sitting in /usr/bin. So many useful little utilities.

I think the same thing happens to a lesser degree with vim features.

A very boring, but instructive way is to read manuals cover to cover - for example the GNU Coreutils one [0]. This way, you become aware of the existence of a lot of tools (such as comm, which is part of the coreutils), and while reading you'll realize that some of them might be a better way of doing things than how you're doing them currently.

Other boring, but instructive reads (heavily GNU biased): diffutils [1], findutils [2], the Bash manual [3].

[0]: https://www.gnu.org/software/coreutils/manual

[1]: https://www.gnu.org/software/diffutils/manual

[2]: https://www.gnu.org/software/findutils/manual/find.html

[3]: https://www.gnu.org/software/bash/manual

The chapter on shell utilities in the POSIX standard is another goldmine!

I'd also recommend Unix Power Tools. There's definitely some outdated content, but I learned a lot about shell usage and text manipulation from it.

http://shop.oreilly.com/product/9780596003302.do

Seconded. The dead tree copy I purchased in the mid-2000s is the best technical book in my collection. I’ve read most of it and every so often I still dip in to learn something new or – more often than not – to re-learn something I’ve forgotten.

I tried out your command and got different line count compared to the output of the commands mentioned in the post. The man page for comm says it assumes inputs to be pre-sorted. To do that, you need to sort the input files before passing them to comm.

  # show only items in both a and b
  comm -1 -2 <(sort -u a_list) <(sort -u b_list)

  # show only items unique to a
  comm -2 -3 <(sort -u a_list) <(sort -u b_list)

  # show only items unique to b
  comm -1 -3 <(sort -u a_list) <(sort -u b_list)

Yes, pre-sorting is necessary for comm to work. I didn't mention that because it's necessary for the commands in the post to work too.

Can also use

    comm <flags> <(command 1) <(command 2)

To use the output of a command as input. This also works with diff and other commands.

More accurately, it can be used for any command which expects a file and doesn't do anything too weird in reading it (e.g. doesn't seek to the beginning and read it again)

The '<( ... )' is just giving the path to that command's stdout as a file descriptor.

    $ ls <(echo hi)  
    /proc/self/fd/11
    $ vi <(echo hi)
    # opens vi with 'hi' as the contents

Thank you. I was just thinking of looking into how it was implemented.

It is called "process substitution", and can work the other way around:

http://www.tldp.org/LDP/abs/html/process-sub.html

With fish:

https://fishshell.com/docs/current/commands.html#psub

But there it only works one way:

https://github.com/fish-shell/fish-shell/issues/1786

Note that this is a bash feature; other shells may have different syntax.

It's the same in ksh at least, so one is covered in the BSDs. It could be also implemented as a separate tool, but not as convenient, because of additional quoting - I did once.

I use it in my frugal SSG project [0] to remove HTML files for every removed source file. It's great to use it this way as it doesn't use temp files.

[0] https://gist.github.com/hadrianw/060944011acfcadd889d937b960...

Indeed comm will work much faster than uniq in this case, because it reads each file only once, and does a single pass on each file.

What the blog post does (doing cat multiple times, and then counting the occurrences) is going to be much slower, although mathematically correct.

Doesn't uniq only read the input once? I've always assumed that the reason uniq assumes its input is presorted is that, then, it doesn't need to buffer all of it: instead, it just eliminates successive duplicate lines.

Yeah but in this case the author is doing "cat 1.txt 1.txt 2.txt".

comm is probably the most useful command I never remember how it is called (and apropos doesn't help).

The only options you usually need for comm are -1, -2, and -3. It takes two arguments, call them 1 and 2. Its job is to include lines from each and both files in output (by default, you get 3 columns of output), so we want to filter by telling it to exclude what we don't want. My mnemonic is "-" means "not", so -1 means "not only in file 1", -2 means "not only in file 2", and -3 means "not in both". So now, when I want comm output that shows the lines unique to my 2nd file, it's:

comm "not only in 1" and "not in both"

so:

comm -1 -3 file1 file2

I know how to use 'man', what I mean is, I never remember the command name (comm) because I always remember it's a command to compare so I think of cmp, and not comm.

Reply on news.ycombinator.com