Ask HN: How do I become a shell god? | Hacker News Reader

Ask HN: How do I become a shell god?

35 pointsh-132 comments

It seems many things can be done with a one-liner or short shell script, but I'm not fluent enough to recognize those patterns yet. I do know a few basic commands, but much of *nix seems like a bag of undocumented tricks; and man pages are hieroglyphic IMO. Is there a cohesive path to learning all the tricks? All of them!

View on HN

Read the whole POSIX specification for the "Shell Command Language", and in particular how input is processed by the shell through all the multiple expansion passes.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...

From there, you have a better background for reading the documentation of shell implementations, like Bash.

Also study Awk:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/a...

Those who don't know Awk do comical things with cockamamie combinations of cut, sed, grep, echo ... and clumsy loops written in shell.

Other than shell syntax and semantics, it's a matter of knowing a large number of utilities.

I find Perl to be good for these things.

That is, or was, Perl's purpose, right? Process pipelines of text data but in a more structured way?

If you encounter a problem or task, then do it using shell commands. This will push into asking the question "what command can do X?" and "how do I connect command X to command Y?" and "how do loop commands over output?". Then just keep doing that over and over again.

Learn the shell chainsaw: `xargs`. `xargs -L` vs `xargs` and `xargs -I` (examples will say use `{}` but I'm partial to `xargs -I @` because that works with Fish and Bash, plus it's a single character.

I enjoy intentionally avoiding some uses of the chainsaw by instead piping into "while read -r", and hence getting a loop body with some room to manoeuvre.

Can you explain or link further about the “chainsaw?” Google comes back with this post.

I think they just meant that xargs is a really powerful versatile tool, like a chainsaw. But also feel free to look up one of my favorite talks on YouTube... Just search for "Unix chainsaw" and it should come up

My way is:

Try to write small loops and pipes on the command line. I do things like this all the time:

  for i in foo*.txt; do mv $i OLD-$i; done

You'll find this breaks sometimes, so use control-r to recall the command and fix it

  for i in foo*.txt; do mv "$i" "OLD-$i"; done

if it gets too long, copy/paste it into a file like ~/bin/renamer and multiple lines and indenting and make it a more permanent part of your collection.

Also, shell isn't the end. I move to python when shell scripts get cumbersome. python can easily parse arguments (argparse), handle filenames with weird characters in them (os.path), walk trees of files (os.walk) and more.

Even better - in BASH and ZSH you can open up your preferred editor and compose a long command in that with "^x e" (or CTRL-x, e) - saving and closing will print the contents into your prompt, ready to be executed with an enter.

I'm often hesitant to run stuff like this because there's a non-zero chance you'll lose important data or bork your OS install with a bad command or typo. I've considered writing an alias that dockerizes my current directory and opens a shell in the new container to test risky commands. Like a localhost staging environment.

Hi,

As of yours loop, such thing, when renaming files in bash, you can do `mv {,OLD-}foo*.txt`

Great practice would be to write report scripts which aggregate multiple data sources into single array of predefined objects which are finally sent into `stdout`, `/var/spool/user` or anywhere else.

> As of yours loop, such thing, when renaming files in bash, you can do `mv {,OLD-}foo*.txt`

I don't think so. Doesn't this break when the glob matches more than one?

Best is to identify the shell as a solution within your daily work. The maybe obvious step 1 is to switch to an unixoid OS such as Linux or Mac OS. Step 2 is to make the shell easily reachable, for example with buttons to open a shell in your current directory or with a drop-down terminal like http://guake-project.org/

Then, whenever you have some everyday problem (frequently some "batch" or "bulk" job operating on many files), consider using the shell instead some 3rd party tools.

If you really are into it, consider working with suitable files. For instance, drop office files (MSO or LibreOffice/OpenDocument) and instead switch to plain text files (Markdown or Latex). They are much easier to process in the command line. Also consider using git instead of dropbox and friends, because it is CLI-first. There is a universe waiting to be explored: There are CLI clients for almost everything (chatting, mail, web, etc.).

I’d recommend the Advanced Bash Scripting Guide https://tldp.org/LDP/abs/html/ and make peace with the fact that you’ll never know it all and enjoy the journey. Years later you’ll still be stumbling across new stuff.

Unix evolved in an age of batch processing.

Files were the default datastore, and text streams the default inter-process interface.

Although the world is changing, a surprising amount of it still operates through those tools.

A good way to understand the tooling is to do one of the Linux from scratch or similar projects, perhaps look at swapping out some tools for self-written alternatives, and scratching your own itch building projects that solve problems you have yourself.

Most people who reach a degree of proficiency in shell also have an innate understanding of the kernel's general organization, the nature of filesystems, the boot process, database types, are capable of programming in multiple languages, comoprehend network programming and the challenges of locking and distributed state, and so on. You can't learn all of it in one go, so just leverage that interest and enthusiasm and devote the time.

Use it a lot and find ways to learn how it works:

- read the manual (see other posts) - customize it (and persevere) - make it fun

The second is really helpful. A lot of my shell commands/utilities [0] are simply better-for-me ergonomics on commonly available commands. These include things like `fzf` wrappers and more sophisticated aliases.

I specifically found the making of `fzf` based utilities to be a source of major learnings. It requires you to know quite a bit of shell mechanics to do things nicely. It's very similar to building a UI with APIs. You invariably learn the surrounding concepts (HTTP, auth for APIs, content types, caching, etc.)

Shameless plug:

I created a tutorial on Linux Terminal Tools a while ago where I covered many topics but not in too much detail. Perhaps going down the rabbit holes on these slides might help: https://ketancmaheshwari.github.io/pdfs/LPT_LISA.pdf

Getting a solid grip on bash scripting was a game changer for me. From there, you can go several directions: Perl, awk, whatever.

By reading 1 dedicated shell book e.g. "Linux Command Line and Shell Scripting" you'll be at the top 10% of shell connoisseurs.

That said, I challenge the usefulness of deep shell knowledge.

If you're planning on writing 300 lines shell scripts, you're most likely doing it wrong, you should be using a full featured, modern programming language.

That is uninspiring... is totally fine to have long shell scripts, and in itself, is no easier nor harder than any other language.

Worth noting that shell is a programing environment (not language) so is highly dependent on the tools you are using or gluing together to accomplish something.

Handling edge cases through bash is error prone, hard to read and hard to maintain (e.g. shell versions, third party tools, etc).

If you think that python and bash are equally equipped, think again.

Unix Power Tools was a great help when I was learning shell scripting:

https://www.oreilly.com/library/view/unix-power-tools/059600...

I love the fact that shell gods exist :)

Honestly? If you can think of anything that needs to be automated, then write a shell script to automate it!

I recently built a side project using bash scripts only. That helped a lot and was very fun, except for dealing with errors.

You need to use it.

I’d suggest taking any task that you would do in excel, and do it on the command line.

Read kiss linux's source code: The kiss package manager especially.

No gods, no kings, only explainshell.com

You could write your own shell.

Two oreilly books to give you an overview, sorta know the lie of the land.

Classic Shell Scripting by Arnold Robbins

Unix Power Tools 3rd edition

Two additional worthwhile books:

The Unix Programming Environment by Kernighan and Pike

Mastering Regular Expressions by Friedl.

All 4 are classics.

Also try tldr.sh website in addition to man pages. They are easier to grok.

man bash

Good starting point

The bash manual is an okay place to start:

https://www.gnu.org/software/bash/manual/

Almost all of my shell scripts are made up of code-snippets copied from Stackoverflow.

Reply on news.ycombinator.com