Comment by nickjj | Hacker News Reader

Comment by nickjj | Hacker News Reader

Shell scripting for processing text. You can often get so much done with so little code and effort.

Also on a semi-related note, I think as a self taught programmer, it's easy to get stuck on things that seem cool but are just procrastination enablers (I know, I've been guilty of it for 20 years). Like, if you're about to start a new project and you want to flesh out what it's about, you really don't need to spend 5 hours researching which mind map tool to use. Just open a text document and start writing, or get a piece of paper and a pen. It won't even take that long.

I spent about 1.5 hours the other day planning a substantially sized web app. All I did was open a text file and type what came into my head. For fun I decided to record the whole process too[0]. I wish more people recorded their process for things like that because I find the journey more interesting than the destination most of the time. Like your journey of eventually finding message queues must have been quite fun and you probably learned a ton (after all, it lead you to message queues, so it was certainly time well spent).

[0]: https://nickjanetakis.com/blog/live-demo-of-planning-a-real-...

These days it might be better to just learn python. It's cleaner and scales better to complex code. And it's ons most system modern systems available out of the box where shells are available too. Shells are still good for simple oneliners, and knoting multiple processes together, but text-processing involves so many different commands, each with their own quirks, that a consistent simple language is IMHO superiour.

Honestly this applies to Ruby & Javascript/Typescript as well and not just Python. I really don't see the value of learning shell scripting anymore when the newer languages are just was easy to learn, terse, and you can adapt better to changing conditions when needed with libraries.

I often find multi-line Python scripts with `import os` and others that could be a fraction in size (and just as clear) in bash. Even more ridiculous are the times I find a node script (published to npm, even) that is little more than a wrapper on a shell script.

Inevitably someone will read these arguments and think “those are just bad programmers”, but your point was that you “don't see the value of learning shell scripting”. The value is in not spewing absurd code like that. Shell commands are fast and efficient. There isn’t an emphasis on libraries because instead you use tools. Is `grep` not enough? Try `the silver searcher`[1] or `ripgrep`[2].

Are shell scripts the best instrument for every job? No, but no tool is.

[1]: https://github.com/ggreer/the_silver_searcher

[2]: https://github.com/BurntSushi/ripgrep

Just because someone knows shell scripting I'm not going to consider them "bad programmers". I know it myself and I've used it extensively before I learned python & ruby.

My point is that the cost for learning and using shell scripts is just too high compared to just using a modern language that's just as terse and a lot more powerful and flexible. Context switching from one language to another isn't free either.

imo the only time shell scripting was practical was when the only major programming languages were C, C++, and Java. imo even Perl5 is more practical than shell scripting.

Also I doubt that the python program you mentioned was that much bigger than a shell script

For processing text, using Python doesn't really make sense in a ton of cases.

If I want to search a text file for a specific string, why wouldn't I just use `grep "hello" myfile.text`, or if I wanted to do it on a directory of text it's a minor change of `grep -R "hello" .`.

Why would I go through the trouble of opening a Python interpreter, or writing out a Python script to do the equivalent in Python?

Or if I wanted to grab the third column of a CSV, I would for sure just use the `cut` command or maybe `awk` (depending on what I'm doing).

For more complex parsing you can often pipe together a few commands and maybe convert it into a 5 line Bash script to make it a little easier to create variables, etc.. It becomes something you can whip up in 1 minute.

Then there's also more involved text parsing that doesn't require piping a bunch of commands together or shell script glue, in which case it comes back to using grep with its various flags and potentially a regexp. It's a natural fit for the problem and you can iterate on it so quickly.

This 1000x. I put of just getting by with shell script for some years and when I finally decided to get deeper into it, it's magical.

A good series of piped commands with tools available basically everywhere can solve problems you had no idea could be so simple to solve.

have any good resources for this?

Perhaps these are a good start:

awk: - GNU Awk User Guide - https://www.gnu.org/software/gawk/manual/gawk.html#Getting-S...

- Grymoire guide - http://www.grymoire.com/Unix/Awk.html

sed: - Grymoire guide - https://www.grymoire.com/Unix/Sed.html - Official docs - http://sed.sourceforge.net/#docs

  man bash  
  man awk  
  man sed  
  man grep

You can also do most of this stuff with Perl one liners if that suits your fancy.

sed + awk - https://www.amazon.com/sed-awk-Dale-Dougherty/dp/1565922255

awk - https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...

general *nix text processing - https://www.tldp.org/LDP/abs/html/textproc.html

There's a free O'Reilly book from last year: https://www.datascienceatthecommandline.com/

awk is pure magic.

The IBM developerworks articles about this are old (2000!), but still incredibly useful and well written. Start here:

https://www.ibm.com/developerworks/library/l-sed1/index.html https://developer.ibm.com/tutorials/l-awk1/

Also, just get to know your shell.

big timesaver: control-r

basic but useful all over:

    for i in *.c; do cp "$i" "$i.bak"; done

etc...

What does double quotes dollar do in the command?

$i is the variable i declared in the for loop. Quotes just wraps it, so that it's (somewhat) safe if the file has a space in the name

What do you mean, "somewhat"? This looks safe for all I can tell.

Ensures filenames with spaces in them get passed as a single argument, instead of being inadvertently expanded into several.

Quoting and expansion issues are a pain in shell languages...

You know how you should almost always eat your vegetables?

You should almost always quote your Bash variables too. Here's why: https://nickjanetakis.com/blog/here-is-why-you-should-quote-...

It prevents filenames with spaces from expanding into two arguments to the command.

I'd split your first point in two:

- Shell scripting for running commands and managing files

- Unix utilities for processing text

These happen to complement each other nicely. It's not that bash is better at manipulating text than Python, it's that Python makes it painful to invoke commands (like those Unix utilities) and pipe data between them (e.g. https://news.ycombinator.com/item?id=17733865 ).

Reply on news.ycombinator.com