Hacker News Reader

Comment by abiro | Hacker News Reader

My rule of thumb: if the script needs argument parsing or any sort of error handling/cleanup, I’ll write it in Python 3.5 instead of bash.

View on HN

Agreed about arg parsing. One day I decided to figure out how to get Python arg parsing in a bash script. The horrifying result: https://gist.github.com/benkehoe/c469ebb76ad80d45405077e1093...

Not horrifying, that's pretty cool!

Or we are both the same kind of horrifying :D

Dude, that's freaking brilliant. Wanting to steal that one...

I agree with the general sentiment, though I set the bar a bit higher (or lower, depending on your perspective). Bash does okay with positional arguments, error handing, and even loops. You can get a lot done with that, and there are a lot of circumstances where a little bash script is just what you need. Where I draw the line is at arrays. If you're starting to think about data structures, bash isn't the right tool for the job.

Agreed. Not with the programming language per se, but I agree with the general premise.

However, in cases where portability is important, bash is would be my choice. If I can pass someone a bash script and not ask them to also install Python and whatever libs I used, it makes my life so much easier.

Containers work great for me if the target users are also familiar with them

Once I type a square bracket in a shell script, I know it's time to bring out Python. I don't think it's humanely possible to write a test command without shooting yourself in the foot in an exciting new way.

> don't think it's humanely possible to write a test command without shooting yourself in the foot in an exciting new way

That's a weird claim. Could it be that you have never really learned shell programming or read the man pages? Or can you give an example? We have lot of production code with many test statements and I don't remember when we were bitten last time by some "surprising feature" of test.

I agree that test works differently than comparisions in many programming languages. So you need to understand how it is different. And you need to be disciplined about quoting, otherwise surprises will happen when you have user-controlled input and the shell starts to do word-splitting. But if you use shellcheck you will be reminded of quoting everywhere. It's not a problem specific to test anyway.

I have written dozens of of bash utilities at a single company. Probably 5000 lines of bash in total. I still don't know the whole syntax for if/for/case and have to look it up every single time.

I've written hundreds (if not thousands) of bash utilities over 20+ years and I still occasionally have to check the syntax for various bits.

> So you need to understand how it is different. And you need to be disciplined about quoting

Or I can use a sane language that let's me get my stuff done faster and with less opportunities to shoot myself in the foot.

If I'm modifying an existing script, I generally have no problem doing that.

But the tweet quoted in the article rings very true to me:

>The opposite of "it's like riding a bike" is "it's like programming in bash".

>A phrase which means that no matter how many times you do something, you will have to re-learn it every single time.

I and probably a lot of programmers write bash scripts just enough to remember the vague shape of what we want to do, yet still infrequently enough to require looking up the exact characters every single time (-z? ! -z? one bracket? two brackets? quotes? no quotes? something about a semicolon and "then", right? where can you add newlines, again? oh yeah, it's not "end" like Ruby/Lua but "fi" right? what's "else if", again?).

I'm sure if we were working with bash scripts daily, we'd memorize it, but it's just rare and just different enough that it's basically day 1 all over again every time. Shellcheck helps as a safety net, but it doesn't necessarily save me a lot of time and cognitive load.

If I'm writing a new script from scratch and I realize there's going to be some branching or prompts or complex argument handling, I just write it in Python. It saves time and effort, it lets other people on my team more quickly modify and debug it, and it's more predictably portable ("oops, only sh, no bash"). And if it ever needs to do something more advanced in the future, it's way easier to extend.

Or put much more succinctly, by 'kristaps in another comment:

>Or I can use a sane language that let's me get my stuff done faster and with less opportunities to shoot myself in the foot.

It's nice for quickly writing something that's just wrapping multiple commands, though.

One of the big points of linters is that they change developer behavior for the better over time.

Once you get used to using shellcheck all the time your code style starts to match what it requires. You start thinking about quoting and other intricacies, and coding much more defensively.

This happens with other checkers as well - >90%+ of the time the python I write passes pep8 and black cleanly after having used those tools for a year or two.

Using Python how frequently?

The issue is the frequency. Due to not coding in it often, you don't retain it.

> Using Python how frequently?

Probably around once a week.

A linter should make you a better coder, whether or not you code frequently, and also serves as a secondary "belt and suspender" check for code sanity and ease of future reading/refactoring.

> -z? ! -z? one bracket? two brackets? quotes? no quotes? something about a semicolon and "then", right? where can you add newlines, again? oh yeah, it's not "end" like Ruby/Lua but "fi" right? what's "else if", again?

really, most of these questions have only one correct answer. [ ! is never correct, neither is ! [ -z or ! [ -n. one bracket in /bin/sh, two in bash. quote every variable expansion unless you have a very good reason not to. add new lines in sensible places. nobody (sensible) complains that you can't write

  for
    num
  in
    mylist
  :

in Python, so don't write it in shell either.

the problem is there is so much cargo culting garbage. the same problem exists in "C/C++", where people write "int* buf = (int* )malloc(num*sizeof(int))" to be "compatible", or do gratuitous pointer aliasing with undefined behavior, despite the fact that any modern C implementation with any optimizations at all will inline small fixed-size memcpy.

I tend to apply much the same rule as emptyparadise - once conditional logic appears, it's time for a rethink. Part of my rationale for this is that I assume that I'm dealing with the thin end of the wedge - if there's enough complexity in the task to warrant conditionals, there's likely a whole bunch more (or there will be in future) that I haven't encountered or grasped yet.

I'm thus willing to take the time-hit on rewriting in Python, because I'm betting that resulting code will be easier to adapt and easier to troubleshoot as this extra complexity gradually comes into view.

I just can't fathom this as I've never really had compatibility issues/quirks with good old #!/bin/bash. Only issues I've had have been with sed on Mac vs Linux. And I've written thousands of lines with lots of logic, generally breaking statements that become nested into separate functions. Sometimes some things become too complicated for Bash, like parsing JSON, in which case I prefer to break out and call a Python or Node script instead of wrapping my head around jq.

But my scripts are also either for personal use, or for internal use (5 person team) at work, so I know who will be running the scripts, why, and where (which OS/versions). Suppose it's a bit like developing for Apple where you can remain aware of the use (and edge) cases.

Awk is a viable replacement for Bash for scripting. A long established standard and installed on pretty much every POSIX system. It has more robust facilities for handling variables, arrays, regex search & substitution, etc.

awk is a full programming language. But it has gotten little attention in the last 20 years.

If you expect that someone else can read and maintain your code with minimal learning I don't think it's the tool of choice.

But... neither is bash. I am sympathetic to the argument that awk is theoretically superior to bash in almost all ways (and it is of course nearly as available as bash). But... something still makes me reluctant to switch to using it in the cases I write a bash script (which I'm already trying to minimize, as is the OP). I'm not really sure why though.

(Fun-for-mostly-me fact, my very first programming job ever was writing awk. It was a while ago. I certainly don't remember any awk now).

Awk is great for snippets that are meant to be copy+pasted into a shell, and is the only reasonable option sometimes if you're stuck in a Busybox environment, but for almost everything else you'd be better off with Python.

This previous comment of mine goes into a bit more detail: https://news.ycombinator.com/item?id=23242753

Agreed but I point that not all awks are gawk.

Or better yet, condition yourself to hit [ a second time and use Bash's built-in conditionals instead of test and [

Depends. In an embedded system I prefer not to have to install Python at all.

In an embedded system (== very limited resources), wouldn't you want to avoid any interpreted language, even shell? We have some very nice languages that compile to machine code these days

BASIC is 56 years old. Computers back then were not very fast, but interpretation was still viable.

Have used it on 7KB RAM :) But to be fair the interpreter was in ROM, no idea how big that was.

ROM was usually very slow, so all ROM content was usually copied to RAM before execution. RAM size was therefore still an issue.

I am pretty sure the PET-2001 didn't do that. 8 KB was all it had, about 1 KB was used by the operating system and 7 KB where available for programming. Actually something you would call byte code today, every basic command was 1 Byte. Just don't use labels with 4 or 5 characters, that wastes a lot of space!

The first models came even with 4 KB RAM / 3 KB usable. Have never seen one of those, though.

If we're talking Linux embedded and not something like an MCU, then you already likely have a shell installed. So you might as well use it for the things which shell excels at. Shell scripts are much smaller in size than the equivalent C code, for those things that shell is typically used for.

Well, there is embedded and embedded embedded :)

I was talking about embedded Linux here, so 512 MB RAM is not really uncommon, many systems have even more. No problem to run even bash. dash is sometimes preferred for license reasons, busybox ash is smaller, but then programming can probably get more annoying already, because useful features are probably missing. I use dash regularly for scripting (interactive use is a pain), haven't had to use ash for a while.

Of course there are much smaller embedded systems. But I don't think they would run Linux.

Rob Landley is working on implementing a full Bash clone (called toysh[1]) in his Toybox project[2]. I'm excited for it to be released - it will be a huge improvement vs. Busybox ash for embedded environments. We'll be able to write scripts that use arrays!

[1] https://github.com/landley/toybox/blob/master/toys/pending/s...

[2] http://landley.net/toybox/about.html

Well, it depends. I had a co-admin who used to rewrite all my Bash scripts in Python. Most of them were a bunch of os.system() calls. He was a bit upset when I pointed out they are actually buggier than their Bash originals and noticeably slower.

Indeed. Shell scripting can easily get overcomplicated and I agree that moving to a real programming language is better at that point. However, Python isn't necessarily the ideal language for it either, because managing child processes with its standard library functionality can get overcomplicated and verbose, particularly once you start dealing with functionality like pipes. So people often resort to generic os.system calls instead and then you get a strange hybrid style.

It's too bad (for bash users) that bash never bothered to embrace ksh getopts. The equivalent of the article's two pages is roughly,

    usage=$'
      [-1][+NAME?script - Script synopsis here]
      [f:flag?Some flag description]
      [p:param?Some param description]:[string?description]
    '
    while getopts "$usage" opt
    do
      typeset opt_${opt}=${OPTARG:-1}
    done

Why Python version 3.5 specifically?

I work on Debian or derivatives mostly and the oldest Debian LTS has 3.5.

Exactly. Or even stricter. I've finally made a self-contained helper for that: https://gist.github.com/fillest/8d64f8fa0cdb1745bfc9c683cf39... Pure Bash is really not needed since we have Python 3.

Assuming you can afford introducing the Python dependency.

Also pipe/subprocess heavy scripts often become sad complex monstrosities in non-shell languages.

So true. In my case, if my script has loop, I will use python or perl instead of bash.

If the script must iterate over the lines of its input, I've found perl -nlE to be a godsend. Basically it is equivalent to

  while(<>){
      chomp;
      <insert your -E code here>
  }

Reply on news.ycombinator.com