Hacker News Reader

Hacker News

Comment by eb0la | Hacker News Reader

Parent Full thread

eb0la

You just need to master a few commands:

Find (take a look at -exec option)

Cut (or awk)

Sed (for simple text/string substitutions)

Xargs

Dd (that beast can do charset transtation from ASCII to EBCDIC to use in mainframes and can also wipe disks)

And bash of course :)

View on HN

One fun thing is that xargs supports parallelization.

    -P max-procs
    Run up to max-procs processes at a time; the default is 1.
    If max-procs is 0, xargs will run as many processes as
    possible at a time.  Use the -n option with -P; otherwise
    chances are that only one exec will be done.

    -n max-args
    Use at most max-args arguments per command line.
    Fewer than max-args arguments will be used if the size
    (see the -s option) is exceeded, unless the -x option is
    given, in which case xargs will exit.

So here is a "map/reduce" job which takes the log files in a directory and processes them in parallel on eight CPUs, then combines the results.

    find . -name "*.log" | sort | xargs -n 1 -P 8 agg_stats.py | sort | merge_periods.py

And for cases where xargs parallelism isn't flexible enough, there is GNU parallel.

xargs is one of my favorites, it's so easy to split something apart to speed it up by a few multiples.

gotta SSH into 100 machines and ask them all a simple question? xargs will trivially speed that up by at least 10x, if not better, just by parallelizing the SSH handshakes.

But that approach is potentially dangerous. See:

https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_w...

"master a few commands"

"awk"

brb 15 years

Might be true. The sed&awk from O'Reilly still lingers on my desk from time to time. And I bought it 20 years ago.

Not really. awk is pretty simple, basically I just assume it's plain C with a huge regexp pattern 'switch' case at the top level, then I google again for the arguments to substr() :-)

Brb in 20 years then.

Some of these have come up for me recently, particular sed, awk, and xargs. Xargs in particular is really gnarly and opens up a lot of possibilities from the command line when used in conjunction with other tools. Love it!

Not a program, but another fun bash thing I've learned about recently is brace expansion: https://www.linuxjournal.com/content/bash-brace-expansion

'man' is probably the most indispensable.

That sounds interesting. Is it documented anywhere?

No, it's something neckbeards pass along to junior sysadmins, and it's hidden very deep in /dev/null.

I find 'apropos' pretty indispensable, because it lets you find the thing that does the thing you need, so you can read its man page.

awk is so beautiful. A good awk one-liner can beat fancy pandas all day long in some cases.

I hate how awk gets so much praise for one liners when it's so good at building things like state machines too: https://two-wrongs.com/awk-state-machine-parser-pattern.html

I've got some similar ones for work that take data csv like data and output sql. A little bit of vim-foo on the csv (yy10000p) and I've got all the test data I need.

If anything, awk should get some praise for it's flexibility to go from one liners to Doom-style 3d games[1]!

[1] https://github.com/TheMozg/awk-raycaster

An example here would be nice.

Don't forget grep!

Also, 'parallel' if you've got any idle cores on your infrastructure.

And perl -pe or -ne ! :)

^ the moment you do anything even slightly non-trivial with sed, abandon it and use perl instead. for even simple regexes with a couple non-"match this char" segments I regularly see almost 100x speedup, it's rather crazy. it's also a LOT more portable in my experience because the regex engine is so much more capable and consistent, even across fairly major perl versions.

Honorable mention to jq. Also get comfy with Unix pipes and streams.

e.g. testing in CI a JSON file is properly formatted:

  jq . < $f | diff -u $f -

Personally, I'm not too interested in any commands that take string arguments in their own homegrown programming language[0]. I'd rather use something like this[1] than jq, to get some extra ROI out of one of the languages I already know.

[0]: https://stedolan.github.io/jq/manual/

[1]: https://github.com/antonmedv/fx

adding socat into the mix if you do network related things! it’s amazing what you can do when you replace bits and pieces of a pipe chain with network endpoints

Reply on news.ycombinator.com