> However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on.
From: http://www.oilshell.org/blog/2018/01/28.html#i-dont-understa...
The consequence of this is that many of my projects which require these overlapping domains (think stuff like build automation) ends up being the two interleaved— either an outer shell script that calls small Python programs (for parsing, URL fetching, and basically any error prone stuff where sane recovery/cleanup may be necessary), or it's a Python program that renders little templates of bash into strings and passes them to bash stdin.
Or in some cases, it's CMake that's the outer orchestration piece, using configure_file to fill templates and then invoke them either at configure time (with execute_process) or at build time (with add_custom_command).
This hits the nail on the head for me. What I'd like is something bash-like to handle running programs, pipes, tab-complete for file names, and so on, and something like Python syntax for control flow and strings (quoting / escaping), and access to something powerful like Python's standard math libraries (and ability to import other stuff like requests). I don't know how you'd roll all that into a single shell, but as close as you can get is what I'd like to see in a new shell.
(I've briefly looked at Oil before, but it has seemed a bit complicated to merit trying to switch full time at this point. I'm definitely following it for future developments though.)
Edit: just found Xonsh from another user's comment, which seems almost exactly what I'm looking for, if a bit hacked together at first glance. Going to try that out.
This is true, but when I am writing a script in Python I generally don't need to pipe as often as I do in bash. Python offer many replacements for things that in Bash you need to call another process.
That is, of course, if performance isn't necessary. When it is bash is still unbeatable because of the pipes.
True, you're never going to pipe something to grep in Python when you can just capture the output and use the inbuilt regex capability.
That said, it can be surprising how much worse performance-wise Python can be than shelling out to external utilities. Consider the simple case of downloading and extracting a tarball. I found that check_call("curl path/to/thing.tar | tar -x", shell=True) was significantly faster than anything I could figure out how to do with requests streaming, httpx (async), and the inbuilt tarfile module.
import subprocess
subprocess.check_call("command1 | command2", shell=True)
Use shell for what is really good for: concise DSL for running commands (one-liners). Leave complex logic (branches, explicit loops) for sane general purpose languages such as Python.
YMMV but for those rare cases when I care about the shell injection, I just don't use shell and run the command directly in Python. Combination of shell one-liners and the main logic in Python works well in practice.
There are many different use cases related to running processes—it is natural that different solutions may be preferred for different cases (check_{call,output}, with/out shell, run, Popen, pty/PIPE, threads/Asunción—all may be useful. And it is just stdlib ).
> I encountered a nice blog post, Replacing Shell Scripts with Python, which, in my opinion, inadvertently proves the opposite point. The Python version is longer and has more dependencies. In other words, it's more difficult to write and maintain.
The Python version uses only the standard library, so it doesn’t make sense to worry about dependencies if you aren’t counting every program your shell scripts call similarly. Having just one thing providing a consistent baseline beats learning that your script needs to upgrade RHEL to get a feature which would make something safer or easier.
Similarly, the Python script is longer because it does better work and consistent, useful error handling. That adds lines of code but it’s usability and correctness, not overhead.
It’s similarly incorrect to treat line count as a proxy for difficulty, as anyone who’s ever had to deal with quoting or data structures in shell scripts knows. The use of pathlib or os.path sometimes gets grumbles for extra characters but I’ve found it nearly inevitable that the grumblers will have something fail or destroy data because they hit a filename or argument with a space or special character in it. On one notable occasion, that resulted in an `rm -rf something something /`. (If you’re writing a shell script, shellcheck is mandatory)
The best criticism I’d make is that it could be easier to replace a shell pipeline. Python’s subprocess makes that pretty easy but you still need a minimum of 3 calls per command plus whatever you need to actually do with the output. Since subprocess.{run,check_output} handles most of my needs this isn’t a big deal and there are excellent modules if you don’t mind dependencies.
The Bash script actually has more dependencies, it relies on a number of external programs (ps, kill, mkdir, sort, ls, cp, echo). What versions are on your system? What versions on the systems of the people running the script? Do they support the same features? If you're running on a Mac are you using the Mac-shipped programs or the GNU coreutils from homebrew?
Also, that Bash script that the author is defending has a subtle bug in it, because Bash is subtle. The bug is here:
$APP $filename >"${output_dir}/summary_${name}.txt"
if [ $? != 0 ]; then
echo "Error $? in app"
fi
Which is exactly the kind of crap that everyone's talking about when saying that we should avoid Bash scripts. It's full of landmines like this.
Bash is great for interactive work on the terminal. It is not great for writing correct, maintainable programs.
About 15 years ago I switched to Python for anything which doesn’t fit on a single screen or uses any advanced feature and have had zero reasons to regret that decision. It’s especially good for anything you use with other people since you spend your time talking about features rather than how to trick the shell into working correctly.
Ruby has pretty good thing wrappers around running shell commands (e.g. using backticks to run a command and get the output as a string or calling system() on a string to execute as a command and get back the error code), and my understanding is that it doesn't really have great Windows support. My impression here is that the author of this FAQ is probably not super familiar with Ruby based on the way they seem to equate the abstractions in both languages.