Do I need to manipulate rich data structures like hash-maps or nested lists? That sort of thing tends to stretch the capabilities of Bash to its limits and I tend to set the bar fairly low here.
Is the program oriented around commands? If I'm gluing executable scripts and binaries, using bash is often superior to a scripting language. Argument passing is more natural and convenient and the built-in support for the standard i/o streams makes it easy for the different commands to pass data between them. The number of variables or lines of logic is usually less important than the number of different installed commands I need to combine. (Or the number of variations on a single command)
Do I need to modify the environment in a significant way? In a bash script it's trivial to source in an environment script, which can be done conditionally or even interactively. This is usually more tedious to do in scripting languages.
Yes, you will have a few extra lines but it will be vastly more readable and maintainable.
And yes, I know I will get the standard the person who wrote the script did a bad job but at some point it should be okay to blame the tools instead of the workman if workmen disproportionately create worse results with a set of tools.
For scripts written in Python, I'd use a similar argument and suggest every serious Python scripter to learn Python. As for Perl, or Ruby, or Julia, or anything really. It's just that learning bash from its man page is, IMO, much easier than learning any of those languages to the same degree.
Of course – and it goes without saying — that some things are just not suited to bash, and in those cases a suitable language/framework must be used and learned if not learned. As far as process calling, environment management, or stdio streams management are concerned, bash is better suited than (all the other languages I've tried) Python, Ruby, or Go.
As a test, run shellcheck on any random shell-script written by these Coworkers from Christmas Past (or your own past) and it will spew serious warnings on almost every single line, run an equivalent analyzer on an equivalently unserious python file and you might in bad cases get 2 or 3 minor warnings per 100 lines.
The comparison is a bit unfair because just by getting these compiler errors and exceptions from a real language you force yourself into a more serious mental mode of programming instead of happy scripting, but that is just another argument in favor for not using bash IMO.
1. People who don't want to consider themselves "serious bash scripts" shouldn't be writing non-trivial bash scripts unless they're okay with it turning out buggy. I agree this is a personal standards thing, and reality is often less simple and more lenient than that.
2. The "compiler errors and exceptions" that you speak of in regards to Python also have equivalents in Bash. Agreed, they're still optional and non-"serious bash scripters" don't often know of them. Which is why I make my coworkers use them when I review their code.
3. There are classes of bugs that would happen in Python (and Go, from my experience) that wouldn't happen in bash, simply because they are in areas where bash shines. At work, I've seen process management and stdio management bugs — some of which have bit us in the field — simply because the non-bash language (Go) has a weird affinity to its child processes, or it (both Python and Go) defaults to not wiring up the stdio of child processes. In the latter case, the proper thing for the developer to do was the same as with bash: read the documentation. Most of our process management code is now in bash (because it's simple and safe) and systemd (because it's thorough and absolute).
But they know how to accomplish this task interactively in the shell, so scripting what they're already doing (or already know how to do) seems like the natural next step. So you wind up with an imperative shell script that's basically a long, flat sequence of commands with some logic and variables sprinkled in haphazardly as they realized they needed it.
Due to the organic way these scripts often emerge, it's not like advocating Python is an easy sell. By the time they think to consider alternatives, the shell version already exists.
If it's just a bunch of cuts or tr, sure.
Would the following be annoying for you to deal with?
#!/bin/sh
sed 's/#.*//' \
| sed 's/:/#/g' \
| cat AMD64 - \
| ./qhasm-ops \
| ./qhasm-regs \
| ./qhasm-fp \
| ./qhasm-as \
| sed 's/%32/d/g' \
| sed 's/%raxd/%eax/g' \
| sed 's/%rbxd/%ebx/g' \
| sed 's/%rcxd/%ecx/g' \
| sed 's/%rdxd/%edx/g' \
| sed 's/%rsid/%esi/g' \
| sed 's/%rdid/%edi/g' \
| sed 's/%rbpd/%ebp/g'
source: http://cr.yp.to/qhasm/qhasm-20061116.tar.gz qhasm-20061116/qhasm-amd64
And I would make an argument though that even large shell scripts in bash have their place.
I often write scripts in either Node or Python, but only when I need things bash is bad about (any sort of proper data structure beyond strings or arrays).
But there are just so many things bash makes insanely easy, especially with operating on files and directories.
And functions that are used as completions or need access to aliases or functions in the current process are also better in bash.
I wish there were a scripting language like bash, but enhanced with at least some hash maps and proper array manipulation, and maybe some formal IPC to allow scripts to request info from the parent process.
As someone who has inherited thousand-line shell scripts, and had to debug many 3rd party scripts, I stand by my assertion.
> Yes, you will have a few extra lines but it will be vastly more readable and maintainable.
Readability is important but it's not the only aspect to maintainability, nor is maintainability to sole concern of a tool. A low bug rate helps maintainability and actually having the features you need, in an acceptable timeframe, is also important.
For example, the OP mentioned the 'set -e' option that causes the script to exit if any command returns a non-zero exit code. In Python, you'd either have to remember to check the return code for every subprocess or define a wrapper, which adds complexity, reducing readability and can lead to bugs and errors. Nor is Python always the best answer for readability anyway. In many cases, it's not like it's just a few lines you're saving. Here are some functions I've used when scripting in Python
import subprocess, shlex
def process_run(cmd_string, stdin=None):
return subprocess.Popen(shlex.split(cmd_string),
stdin=stdin,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
def process_results(process_object):
(stdout, stderr)=process_object.communicate()
return (process_object.returncode, stdout, stderr)
def process(cmd_string, stdin=None):
return process_results(process_run(cmd_string, stdin=stdin))
f=SpooledTemporaryFile()
f.write(stdin_string)
f.seek(0)
results=process(cmd_string, stdin=f)
f.close()
return results
Actually what I'd say first is that it's quite possible the person writing the script knew what they were doing. I've inherited bad code in my life, I've inherited some real gems, and I've inherited a lot of code in between. One thing I've learned is that I tend to be unfairly critical of average code. It's hard to read unfamiliar code and easy to criticize inconvenient design choices when you have to adapt their code to some new problem that they never anticipated. Usually I'll be better off just buckling down and untangling the spaghetti.
Also, who is to say what I should be using "99% of the time?" Each problem has different constraints and different priorities.
Is there a reason you cannot just say `#!/usr/bin/env python3` in your scripts? I don't see why you require python3 to be the default, am I missing something here?
Again, in the context of this discussion, the whole argument is yet another point in shell's favor. There's no major backwards-incompatible change in the language. With Bash, you just decide whether POSIX compliance is something you need, and that's basically it. Both versions are still supported and no one interrupts discussions to announce that beatings will continue until morale improves whenever the deprecated version of Python comes up.
Or you could put the wrapper functions in a module and call it a day, either way problem solved.
> Do I need to manipulate rich data structures like hash-maps or nested lists? That sort of thing tends to stretch the capabilities of Bash
> If I'm gluing executable scripts and binaries, using bash is often superior to a scripting language
The two points above resonate with my view that there is a missing piece. On one hand we have bash which is optimized for being the glue (second point). On the other hand we have Ruby, Python, Perl, Go, etc which are good for first point. What I think is missing is newer, more powerful shell, which supports both use cases and more. I'm working on it:
Please note that I'm not the only one that thinks there is room for more powerful shells. See the readme for links to other projects.
Bash scripts are slow as hell (as most commands have to spawn new processes), it is hard to write "secure" code (if even possible), handling whitespaces can be a pain in the * and the amount of repetition is awful. If your kid has done something wrong, just tell it to write a bash script: it is the equivalent of writing a hundred times:
x="$(...)"
It is the glue which keeps everything working together.
Disclaimer: Please don't build a complete cathedral out of glue.
--JepZ
Doing things properly in Python or Go may take a few more lines (not much more really) but it is 100 tones more robust, and you need that if you are writing anything more than a 10-line one-off hack.
Exit codes are a poor substitute for proper error handling with verbose error reports. They do the job most of the time, as long as you remember exactly which command behaves which way. And that's a clear path to mistakes :-(
I certainly prefer Perl for large scripts (Python is good too but I happen to know Perl better), but I often have to write scripts for targets that have bash but don't have Perl or Python. Since I've been forced to use bash, I've found it to be a better scripting language than I expected it to be.
You pretty much have to put all the init stuff in a bash script. I can't think of any real-world examples of userdata scripts being less than 10 lines.
The alternative of course would be a three line script that downloads a file and executes it. But what language would that file be in? Probably bash. And it would make the infrastructure-as-code tracking and deployment process much more complicated.
You can use Go/Rust for static linked bootstrap instead of Bash. (See rustup.)
I'm thinking about combining GitLab (private token access + repository files: https://docs.gitlab.com/ce/api/repository_files.html#reposit... ) and a bootstrap script in Bash, that then launches something better. Plus self registration back into a GitLab repo (hey, it has the access token already, so it can push).
Yes, Ansible/SaltStack/Chef/Puppet reinvented, but you're now not bound to the horrible database, language, bootstrap, speed (slowness) and workflow of any of them.
If you want more access control, then creating a HTTP microservice that has the real private token (or proper OAuth2 access) to GitLab plus handles the self-registration and token handout/revocation is easy compared to the other parts. (And it can also store everything in GitLab.)
However, note that userdata will only run once at launch time. If you ever need to re-run userdata, you have to stop the instance, remove the userdata from settings, and then add again. YMMV.
I do the Python way when launching a beanstalk instance (basically cloud init).
At the least, rather than Bash, you might consider Perl as a default, lowest common denominator for scripts that need to run anywhere.
- It's nearly as ubiquitous as bash.
- It has approximately the same kinds of file/path operations built in.
- It has reasonably good support for strings/regexes/etc. all built-in, so you don't have to call out to tools like sed/awk/grep all the time and hope that they are available and compatible across your target platforms.
- It provides reasonably good arrays and hashes, which are horribly horrible in bash.[1]
- You can use syscalls very easily if you really need to, but usually you don't.
[1] Of course, no language can save you from the file system disaster (https://www.dwheeler.com/essays/fixing-unix-linux-filenames....), but being able to know that "foo bar" is a string instead of two array elements is a good start.
Mostly this all applies to Ruby or Python too, modulo perhaps the degree of ubiquity.
Perl regexes are the best of breed that everyone else replicates — far better than “reasonably good.”
The Perl erasure in this HN thread is startling.
"Erasure" to me implies some active effort to remove Perl from discourse. I don't see anything like that here: indeed, there are a number of positive mentions, and no negative ones I see. Granted, Python and Ruby are both mentioned more often, but none of those is at Perl's expense. Am I misunderstanding what you mean by 'erasure'?
I suppose being ignored beats the typical herp-derp anti-Perl bigotry, but I’d prefer all-around civility.
I'm having a hard time following you here. Do you think that they're doing so for any other reason that Perl is no longer their go-to tool? There are communities where Perl is still used: PostgreSQL for example uses Perl for some of its scripting, as well as its build farm tool, in particular because of its portability on older systems.
That said, from what I've seen over the past 10 years or so, Perl hasn't had much of a presence in areas where a lot of computer work in tech is being done. For example, in cloud computing, or scientific computing, or machine learning, or web frameworks. Please don't read this to mean that Perl couldn't be or isn't being used in these cases or wouldn't be a better fit. (As an aside, I think Perl missed out a lot while a large portion of the community was focused on Perl 6: there's only so much energy in a community, and that absorbed on Perl 6 wasn't focusing on evangelism. But that's not something I'm interested in litigating here.) Or that there isn't something a bit frustrating in seeing the wheel reinvented time and time again. And so many examples on the web use bash as a common denominator. This puts Perl further out of mind if it's not already part of your everyday workflow. And how many developers today have come of age without seeing Perl in their everyday environments?
Consider the current forum. What's the percentage of front-page posts that are about Perl or tools where Perl is a part of the tool chain? It would be understandable for the people who frequent HN to not view Perl as their go-to. I don't consider it uncivil for people to neglect to mention some other language when it's not something they'd actually think of reaching for. It seems the solution would be to share examples of where Perl provides advantages, both in the comments here and in submissions to HN.
If i had to choose only one of ruby/python or perl i would choose the former and it would be able to cover my base both as glue-code and for more programs. Perl would maybe make the glue code a bit easier but instead i would be much less employable and have a much harder time finding other people who can read the glue. I'm not qualified to have an opinion on perls capabilities for other programs but i'm sure there are valid reasons most people prefer other alternatives.
After using the linux command line (or its many «relatives» like cygwin, mac os, unixes) for more than 10 years now, I’ve talked to exactly one person that used perl to accomplish anything at all. He used it to edit text files, so he could have done the same in awk/sed/vim in my opinion.
I know more people who write fortran 77 than perl.
People just don’t seem to use (or like) Perl very much.
We support a lot of different OSes, and there's usually less variance between the Perl deployed on them than there is in the shell.
The problem is that Bash has about the lowest barrier to entry which can be found: just dump the things you were going to type anyway into a text file and mark it executable. It's simple, and then you bloody your nose on one of Bash's many idiosyncrasies, and people will say, "Oh yes, ']' is just a required last argument to '['. You gotta watch for that." And the true Bash master knows that my rule is silly and that anything may be written in Bash.
Just...please don't.
I think it's a fine guide. If I'm writing something that starts approaching a program rather than a few lines of utility-throw-away, it's time to at least immediately start co-developing in something more sane. My exception is for Makefiles, if one considers them "shell".