Machine learning with Unix pipes | Hacker News Reader

Machine learning with Unix pipes

jott.live

188 pointssigna1138 comments

If you interested in unix-like software design and not yet familiar with kaldi toolkit, you definitely need to check it https://kaldi-asr.org

It extended Unix design with archives, control lists and matrices and enabled really flexible unix-like processing. For example, recognition of a dataset looks like this:

extract-wav scp:list.scp ark:- | compute-mfcc-feats ark:- ark:- | lattice-decoder-faster final.mdl HCLG.fst ark:- ark:- | lattice-rescore ark:- ark:'|gzip -c > lat.gzip'

Another example is gstreamer command line.

If anyone else is following along and getting a generic c++ exception from bun, try upgrading bun using `bun upgrade —canary` to get a more helpful error. I had to update my nvidia driver for ArrayFire: https://discord.com/channels/876711213126520882/104207192669...

Thanks for flagging!

Excellent idea. This one is more usable than the python ones.

At least i don't need to use the horrible Python to do machine learning. Thanks.

Blog post begins with cat abuse. Fyi, grep [ex] [file] is equivalent to cat [file] | grep [ex]. I get that they needed a simple pipe example

    cat /usr/share/dict/words | grep -e '^a..$' | wc -l

can be shortened to

    grep -cx 'a..' /usr/share/dict/words

but yeah, author probably just wanted to show pipe example. I usually show `du -sh -- * | sort -h` for such cases.

Performing 100 of the cat versions on M1 MBP with the default zsh consistently takes between 6.1s and 6.2s, whereas 100 of the grep -cx version consistently takes between 9.1s and 9.3s

grep -cx version:

    for i in {1..100}
    do
    grep -cx 'a..' /usr/share/dict/words
    done

cat version:

    for i in {1..100}
    do
    cat /usr/share/dict/words | grep -e "^a..$" | wc -l
    done

(adding a | tr -d " " to the cat version to make the output equivalent only adds about .1s to the total run).

6 and 9 seconds!! Something seems off. Here's what I get (Linux, Intel Gold G6400 CPU @ 4.00GHz):

    $ time for i in {1..100}; do grep -cx 'a..' /usr/share/dict/words; done > f1

    real    0m0.249s
    user    0m0.207s
    sys     0m0.046s

    $ time for i in {1..100}; do cat /usr/share/dict/words | grep -e '^a..$' | wc -l; done > f2

    real    0m0.306s
    user    0m0.386s
    sys     0m0.141s

Edit: probably because your word file is similar to the one in the post? I've got only 39 matching lines from a 102401 line word file.

235886 lines with 90 matches on whatever is shipped with macos - shasum /usr/share/dict/words = a62edf8685920f7d5a95113020631cdebd18a185

I also put each version in a separate file, so my time commands were time ~/a and time ~/b, and I let the output go to stdout

Also, weirdly, the commands you provided don't print the results from the time command - I'm not sure why (I never use zsh, it's just on the computer I'm using at the moment).

I just tried with 662349 line file having 177 matches ( got the word file from http://app.aspell.net/create). I got `0m1.327s` for just grep command and `0m1.412s` for cat+grep+wc version. I use bash, so not sure about zsh. 6+ seconds really seems off.

AFAIK GNU grep is a different beast than other grep implementations.

This is an unfair comparison because you aren't using the same `grep` options. What your test shows is that `-cx` is not as efficient.

On an x86 MBP:

    $ time (for i in {1..100}; {grep -cx 'a..' /usr/share/dict/words}) >/dev/null
    user=9.22s system=0.34s cpu=94% total=10.097

    $ time (for i in {1..100}; {grep -e '^a..$' /usr/share/dict/words | wc -l}) >/dev/null
    user=6.14s system=0.52s cpu=95% total=6.948

    $ time (for i in {1..100}; {cat /usr/share/dict/words | grep -e '^a..$' | wc -l}) >/dev/null
    user=6.29s system=0.80s cpu=100% total=7.024

I just used the grep command that OP mentioned in their cat abuse post.

seems that OPs suggestion was only meant to make the command more concise

Not equivalent. If you want to insert an intermediary processing stage between the input file and the grep, the initial cat makes that a mechanical insertion, mirroring the intention in syntax. Without the cat, it's a more complex transformation.

Additionally it costs almost nothing to run a cat, but some mental work to remember which programs are built to prevent it (accept filename argument). If they are one can pass /proc/self/fd/0 as filename... That's some mindless tinkering that works so universally that it's just practical

You can reduce the cognitive overhead by using this pattern:

Before:

    <input.txt sed ... | awk ...

After:

    <input.txt grep bar | sed ... | awk ..

Every new pipeline stage is just "command args |" and you insert it after the end of a previous command, treating the initial input redirect as the end of zeroth command

...that increases the mental overhead because now there's no delimiter between the first and second part of the pipeline.

in zsh you can do this:

   <input.txt | grep bar | sed ... | awk ..

but not in bash

Exactly this. I.e. I often back and forth comment out parts of a pipeline, when debugging. Thus, with `cat [file] | grep [ex]` I could do:

    cat [file] | less # | grep [ex]

... to quickly check the output of the unprocessed data, before uncommenting again. This is not possible with `grep [ex] [file]`.

The power of this of course shines the most when having really long pipes, of up to or over 10 parts (which I've had lots of in my career).

(Addendum: Well, I of course actually have an alias s='less -S' to make this even more fluent, so I do:

    cat [file] | s # | grep [ex]

... to peek into various parts of a pipeline. -S is for not folding long lines.)

Ya,ya. Cat abuse is often mistaken for "I'm using cat as a placeholder for some other more complicated process". Get over it

There have been times recently when I've used cat in places where an alternative exists just in the hope that it annoys someone indulging this hang-up.

pv may fit the bill, I use it to get transfer rates for disk images I'm writing

ref: http://www.ivarch.com/programs/pv.shtml

Indeed. I'm genuinely baffled that anyone would care enough to mention it given how common an idiom it is. When did this so-called 'cat abuse' rise to the level of 'I must chastise this person for their sin'?

it's a micro optimization and some people see all of life as an optimization problem.

Insightful. And rather sad. I suspect there's more than a slice of "old == bad" mixed up in this attitude as well.

P.S. - do not search for 'cat abuse' to understand this mentality. Rookie error.

https://porkmail.org/era/unix/award is a good reference if you want to read what the heretics believe about cat and kill and wc -l.

I'll use head instead of cat while testing if my dataset is big, and it's just easier to swap out. it's not cat abuse, it's cat petting

I think the root cause of cat abuse is that grep just takes the parameters the wrong way around. It happens much more often that I want to tweak the pattern and I know which file(s) to grep than the other way around.

I think it's less that the parameters are the wrong way around and more that the file-reading feature in grep is redundant and could be deleted.

Except when you pass `-r` and you pass a directory in place of the filename. But maybe it should have been `-r <directory>` and still no positional file parameter. The ship has been sailed a long ago though.

The newer generation of grep-like utilities (ack, ag, rg) recurse through a directory parameter by default. Of course you could always do this with find.

Anybody knows if some shells apply these easy optimizations? Could a shell detect cat abuse and replace it with the right grep incantation?

cat and grep generally aren't builtins, so shells can't rely on them to have any particular behavior.

Cat could use splice and basically reduce the overhead to nothing. It possibly already does.

> Could a shell detect cat abuse and replace it with the right grep incantation?

That sounds like shell abuse via bloat.

Not saying it is necessarily a good idea, just curious if any such optimization is possible. It seems (from thewakalix comment above) not being a built in it would not be possible.

Now we only need to remove the other 100% of the bloat in the shell ;)

Reply on news.ycombinator.com