Hacker News Reader

Comment by pwdisswordfish8 | Hacker News Reader

I've recently taken to not bothering with shell scripts at all. I just use Python instead. The type system, while it's by no means state of the art, is still miles ahead of the stringly-typed bash. The libraries are excellent; much can be done with the standard library alone. The scripts are much faster (even though the language isn't particularly tuned for performance), since I don't need to spawn a subprocess for each little string manipulation task. And if I ever need to ‘shell out’ to an external process, there's always the subprocess module that is much more robust than ‘set -e’ ever was.

Seriously, if you can avoid it, just don't write shell scripts at all. Use a real programming language instead.

View on HN

This, and I mean very specifically this with python always as the go-to replacement, comes up every single time anyone even utters the words "shell" or "bash" and every time all I can think is that the people saying it aren't writing the same kinds of shell scripts I ever see, use, or wind up writing.

And every time I've ever seen something that's allegedly "a shell script replaced with a python script" it's way more complicated than an equivalent shell script would have been.

I just wonder, really, what "a shell script" even is to you if not something where you need to "shell out to external processes"?

I wonder too. Things like file and directory manipulations are way more complicated in native Python and I find the exceptions harder to troubleshoot than native commands. I'm not even going to get into pythons version and dependency hell, suffice it to say I have more confidence in a standard version of bash and standard utilities generally being present on a modern Linux system. What I've seen from pythonistas as reasoning for this replacememt love is usually one or a combination of a few things - a claimed write once run anywhere ability that I think is generally fantasy, a belief that Python is a "real" language and shell is not which usually just means "I don't like shell or think it is unfashionable", or a belief that Python code can be made more reusable. The person in question almost always has a background of feature developer that is now doing devops type tasks.

All that said I like python and there are tasks I absolutely prefer it for, usually anything to do with scraping some web API that returns a complex piece of JSON or xml.

Here is a novel idea.

I am allowed to call 'mv' or 'cp' binaries from bash.

I am also allowed to call 'mv' or 'cp' binaries from python with subprocess!

(Edit: bad example binaries, 'df' or 'tar' would be probably better.)

I must choose not to install bash libraries for a bash script beyond builtins.

I can choose not to install python libraries beyond builtins.

It becomes a really simple tradeoff.

Nobody forces you to step out of python's stdlib. No extra dependencies. And you can still reap benefits of a proper programming language.

It's worth noting that there are still a shocking number of production machines in the world that don't have stable installations of python3.x yet.

So? Python 2.x is just as good for this, and is pretty widely available. Besides, I don't write scripts for every production machine in the world, just a subset that I have access to.

So, a bash script written ten years ago probably still works on any version of bash since then.

A python script written ten years ago probably doesn't even parse on python3.0, let alone python3.6 (because there have been backwards incompatible changes since 3.0 even!).

And that's also true vice versa.

So like, this is to the point that there's some "reasonable subset" of python you can use to make it portable (ie. no external packages so you don't need to worry about pip vs. setuptools or venvs or whatever, let alone whether they'll build). I'm asserting that there is not.

Not too long ago I discovered that a tiny temporary shell script I gave to a team to validate Akamai cache was still being used nine plus years later. It was supposed to be a short term work around until the UI team created a proper validation. They never did and somehow my script kept working for almost a decade. The only reason I found out was due to a minor change in our UI that broke the manifest lookup url pattern.

Again, so? Maybe you should update your scripts more than once every 10 years?

It is OK to call whatever one needs actually - just minimize the usage of Bash language features (Python provides most of them). It is not complicated and requires no dependencies, here's a self-contained helper: https://gist.github.com/fillest/8d64f8fa0cdb1745bfc9c683cf39...

Great, would that work with pipes and redirections too? At that point you might as well use sh[1] which is a more advanced abstraction. My issue with these approaches is that you'll inevitably hit a bug or limitation where just using a shell language would've been easier to maintain and more portable.

Python has its uses, but replacing shell scripts is not one of them IMO. It it was then the problem didn't need to be a shell script to begin with (doesn't shell out often, needs complex data structures, etc.).

[1]: https://amoffat.github.io/sh/

Python's way of executing a command is somewhat of a nightmare. There are many, many ways to do the same thing, and some are safer than others. Some are better for capturing output, while others are better for just executing and getting a return code.

> file and directory manipulations are way more complicated in native Python

Have you looked at the pathlib[1] module that was added in Python 3.4? It's fantastic. Makes things both more convenient and more correct, in my experience, and it lets you manipulate Windows paths on Unix and vice versa.

[1] https://docs.python.org/3/library/pathlib.html

Hum, no those people are probably writing the same kind of script you do. Their Python versions are indeed longer and more verbose.

I always recommend Python anyway if you care about edge cases (what is not always). That longer script handles them on the obvious way, while the short and more readable shell script does something absolutely crazy every time a detail is different from planed. And if you try to correctly handle the edge cases in Bash (you'll fail), you'll get something much more complicated than the Python version anyway.

Which of these is simpler?

Rust:

    fn add(a: i32, b: i32) -> i32 {
        a + b
    }

Python:

    def add(a, b):
       return a + b

Arguably the Python function is "simpler" because it occupies fewer characters. And yet, the Rust function is more tightly specified; it does less. It can't throw exceptions, unlike the Python function. Invalid programs where you pass the wrong type to the Rust function won't even compile, whereas the Python function will blow up at runtime.

I figure that these supposedly "simpler" Bash programs are not simple at all.

Perhaps sometimes. But properly handling errors around process management in python is not simple either, even with libraries like subprocess. My experience is that when python scripts doing a lot of process management fail they fail in far more complex and hard to diagnose ways than shell scripts do (especially bash scripts with `set -e -o pipefail`).

And like, `subprocess.run()` defaults its check argument (that will make it throw an exception on non-zero exit code) to false, which means that to get that same kind of behaviour as `set -e` in python means making sure you always pass check=False to every invocation, afaik.

Sometimes added complexity can introduce classes of errors.

I'm a little lost... neither of those functions is doing something you would be expected to do in a shell script.

The upthread point was that shell scripts are for coordinating the actions of separate programs run via their command lines and simple IPC like pipes. If that's not what you want to do, shell is the wrong language. If that *is* what you want to do, then you'll find Python and Rust are pretty severely handicapped.

What I was trying to say is that bash scripts are superficially simple but their behavior is hard to reason about.

I chose Rust for my example because its addition operation is tightly specified and thus the behavior is very easy to reason about. (I probably should have chosen a language other than Python for the counter example.)

As soon as shell scripts get mildly complex, like others in this thread I prefer to port them to Python.

Since I'm a FOSS programmer, I've often dealt with the misery of non-portable shell scripts. Userland incompatibilities across operating systems mean that the same bash program behaves differently because the userland programs bash calls behave differently.

In contrast, reasoning about the portability of a Python script is much easier than predicting which tangle of flags in a shell script will do the right thing.

> What I was trying to say is that bash scripts are superficially simple but their behavior is hard to reason about.

But the counterpoint is that this is dependent on situation, and if the behavior you're trying to "reason" about is better captured by the operation of tools like "find", "objcopy", "openssl", "nc", "rsync", etc... than it is by a Python library-based attempt at the same stuff, then putting that behavior into a shell script is a net win regardless of "complexity".

Oh, I'm really not comparing it with Rust. The comparison is between a general language and a shell. If it's Python, Rust, PHP, Java, or whatever is of secondary relevance.

I've done some ops work in C, Python, and C#. I'm also waiting for a good problem to try some Haskell shell monad. All the same old language pros and cons apply.

I'm just making a general argument about what "simple" means. A well-specified program may or may not be superficially simpler, but it will exhibit simpler behavior.

Oh, ok. The Rust version is at least as simple as the Python one, depending on what kinds of detail you care about.

The thing about shell code is that there are plenty of cases where you don't care about things like error management, bad inputs, and etc. So it's not immediately obvious what language is simpler. When you care about those, the shell becomes the most complex choice by far, and that's the duality problem that generates all those different opinions you see around.

> I just wonder, really, what "a shell script" even is to you if not something where you need to "shell out to external processes"?

A replacement for a shell script may not itself be a shell script.

A fairly typical python-as-shell-replacement I have that (among other things) creates/manipulates test resources in AWS corresponding to git feature branches, shells out less than it would if it was a typical shell language (using boto3 rather than shelling to the AWS CLI, etc.) but still shells out to git. But it would be replacing a shell script even if I had a git-repository-interaction library that eliminated the need to shell out.

I mean I'd agree that's a good candidate for replacement; once you start doing anything more complicated than maybe idempotent curls to an external service I think it ceases to be a 'shell script'.

This may sound a little like "no true scotsman" I guess, but "what is the scotsman" is kind of the key issue that I'm saying gets glossed over when people say blanket things like "never write shell, always use python."

I’m not sure I’d agree with defining the scope of things for which an actual shell language is ideal as inherently “shell scripts” (for a few reasons, including the evolving variety of shell languages, the fuzzy boundaries of that set, etc.), I think that it is probably the case that the choice between bash (or powershell or any other shell language) and Python isn’s as simple as “always use Python” and that there is plenty of space for discussion around which tasks each is appropriate for, with the answer differing based on which shell languages you are looking at.

This (and a loooong history of similar posts) is the post that I was originally responding to:

> Seriously, if you can avoid it, just don't write shell scripts at all. Use a real programming language instead.

This may be slightly weaker than "never use shell" or "always use python" but it's pretty close to the former (you can almost always "avoid" using shell somehow), and people seem to believe that python is somehow particularly uniquely suited to use for shell tasks that I think there's some degree of the latter implied.

So... I agree with you? I am saying that there is room for discussion around what constitutes a good use of shell script, and that there is a space in which that is true. I'm suggesting that a lot of people who believe otherwise may just mostly be writing things that aren't in that space, and I think this is actually a pretty charitable argument tbh.

As someone who has gold in Bash on Stack Overflow, I can say this with confidence: Bash is just a shitty programming language.

If you are writing anything larger than about 60 lines, don't use Bash. Almost anything else will be better. Python, PHP, Ruby, JavaScript, even Go.

True. However 60 lines of bash would amount to 300 lines of Python.

Possibly, not necessarily, and I'd argue it's for the better. If a bash one-liner requires a wizard to decipher (especially because people way overuse short options over long ones in scripts), and 10 lines of equivalent Python can be understood by someone who has never actually written any Python, I know what I'd pick.

This also helps in eradicating bugs. Bash is very hard to get right (hence this thread in the first place), even for the simplest stuff. This simply won't happen in Python (or any other proper language).

I agree, but I use Powershell rather than Python. (I've actually written Powershell scripts that implement pieces in Python.) Because gluing together bits of Powershell and Python is still better than trying to write it in bash.

Same. I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Stack Overflow always has 100 answers on how to do this, and every one has someone replying "well that doesn't work if..."

In general, after 20 years of using Unix, I'm starting to sour on the "everything is a string" and "streams are just newline separated plain-text records". The shell is a pretty good REPL, and glue like seq, find, xargs, grep, etc. are very good. But honestly, I find myself reaching for purpose-built tools rather than ad-hoc pipelines more and more. fdfind does more of what I want on average than find; rg does more of what I want than 'find | xargs grep'. I also find myself using more and more structured data, and write more jq pipelines than I do bash pipelines. Finally, I am getting more and more frustrated at basic shell mechanics like history handling. I have 5 terminals open on average, and I don't understand why C-r in one won't find history in another (I know why, of course, but I don't like it). I'm getting pretty close to just writing my own shell and toolchain. I feel like if the Unix shell needed to be fundamentally reimagined, someone would have already done it. But they haven't. They just hacked Unicode and remote RPCs into shell prompts so that pressing "enter" on an empty shell prompt takes 25 seconds to run. I'm not sure why people are so focused on the polish and not the core inadequacies, but they are, and I feel like I'm going to have to fix that myself in the next couple years...

Have you tried any of the shells here?

https://github.com/oilshell/oil/wiki/Alternative-Shells

Discussed recently: https://news.ycombinator.com/item?id=26121592

A lot of them are "fundamentally reimagining" shell -- actually I'd say MOST of them are; whether that's good or bad depends on the user's POV.

Oil is reimagining shell, but also providing a graceful upgrade path. Out of all the shells I'd say it's most focused on the fundamental language and runtime, and less on the interactive issues (right now).

I couldn't agree more!

> I feel like if the Unix shell needed to be fundamentally reimagined, someone would have already done it. But they haven't.

AFAICT this has a lot to do with the fact that most people who'd be able to "reimagine" / fundamentally improve the Unix shell are also the people that are so intimately familiar with the shell and all its quirks that they don't see any need for improvement. It has always been that way, so it must stay that way.

> I feel like I'm going to have to fix that myself in the next couple years...

Please let me know when you do. :)

I agree with some of your frustrations, but it's honestly not that bad once you setup your environment properly and get used to some of the quirks.

> I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set".

Adopt some best practices[1] and have a template to start new scripts with. ShellCheck also helps with avoiding the most common pitfalls.

> I find myself reaching for purpose-built tools rather than ad-hoc pipelines

It helps to distinguish day-to-day interactive work in the shell from writing portable shell scripts that will be robust and require little maintenance for years to come. For the former you want user friendly tools that help you the most with a particular task. For the latter you want to use battle-tested tools with a stable interface that are available on many systems or are preferably built-in to the shell.

> I am getting more and more frustrated at basic shell mechanics like history handling

That's mostly a solved problem in many shells. Zsh supports it natively[2].

I wouldn't want to discourage you from writing your own shell, good luck with that, but there are plenty of good POSIX compatible and alternative shells out there. We can agree that "real" programming languages are much more powerful, safer and friendlier, but plain old shell scripts are the right tool for the job in many situations and shouldn't be ignored at all cost.

[1]: http://redsymbol.net/articles/unofficial-bash-strict-mode/

[2]: https://nuclearsquid.com/writings/shared-history-in-zsh/

All the deployment things are just wrappers to shell commands so if you want to help people debug user-data.sh, cron, ansible, docker files, or puppet, you need to understand shell. Also, it is better to have something in bash as code than not have it be fully automated. I draw the line at arrays tho. If you want arrays don’t do bash. But if you want to just run a bunch of commands on your compute, bash is there.

One thing I have started doing for stuff that need to go via ssh is I just do a here doc for an embedded shell script that I scp to the remote and then just use ssh to invoke it. The quoting and pseudotty stuff just got to be too dumb to understand.

Also, and maybe this is just for non-interactive scripts, I find everything from writing them to using them is a lot easier if you take pains to make them idempotent, so you can run them one or 100 times with the same effect.

> The [python] scripts are much faster...

Pipelines are performant; many algorithms can be expressed as pipelines. Bash has many built-in string manipulation mechanisms which do not require a sub shell.

Why do people who subscribe to the “avoid shell scripts at all costs” ideology feel so passionately about prescribing to others?

I like python, but if I am in a rush it’s shell.

Actually, the one thing that I as a systems engineer really like about Clojure is the threading macro (-> some-data some-function-applied-on-some-data some-otherfunction) or (->> ...) for inserting not as a second, but last argument. That is like a pipeline but usually more readable to me even though I am proficient with the shell (and I must admit even PowerShell, which really is more comparable to Perl with a bit of SQL and other flavours here and there than bash).

Sysadmins/ system engineers might love Babashka: https://github.com/babashka/babashka which is a large subset of Clojure + some frequently used libraries as a native GraalVM image. It is portable, has very fast startup and if the script becomes a larger program, you can easily switch to ClojureScript + Node.js (e.g. for still very fast startup) or Clojure (on the normal JVM) or perhaps build your own GraalVM image. You might also just open a REPL and run it as a single session but that is rather unique in the sysadmin/ systems engineer space, where most things are launched on schedule e.g. each 5 minutes by a script and in case the startup time is somewhat long, it might dominate the execution time.

Btw. babashka seems to be about twice as fast to start on my Debian: time bb -e '(+ 1 1)' executes on average in about 11 ms vs time python3 -c 'print(1 + 1)' executes on average in about 23 ms

> Why do people who subscribe to the “avoid shell scripts at all costs” ideology feel so passionately about prescribing to others?

Are you asking why people make recommendations in general?

Pipelines are a kernel feature that can be created on Python too. You'll just need to abstract it in a function.

At least shell follows POSIX standard which is more than one can say about Python. Have witnessed self-proclaimed "shell replacing" python scripts invoking commands, and never checking the exit code. Debugging hell.

I like bash because it's easy to connect different data together. I can output a column from a sql query to a file and easily write a for loop that goes over the data and use it.

I work mostly in payment related code, so most often I use bash to hot fix in production. Like if we need to refund certain people, I'll convert the refund api call to a curl call, get the data from sql and then just run a simple loop.

Shouldn't scripts over 100 lines be rewritten in Python or Ruby?

http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#sh...

tl;dr I think the main skill that is missing is being able to write a Python script that fits well within a shell script. The shell script often invokes tools written in other languages like C, Rust, JavaScript, etc. (most of which you didn't write)

Good book on this: https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe...

Online for free: http://www.catb.org/esr/writings/taoup/html/

Use shell script for glueing stuff (ie calling other tools) and pipelines. Use GNU awk for complicated text processing. Profit!

Bash is fine as long as everything works. Handling errors in bash is a fool’s errand and you’re quite right to use Python for resilience in the face of errors or malformed input.

Sometimes you just gotta eat. Python is a sit-down meal. Bash is a can of beans. I’m happy eating both for dinner but it depends who (if anyone) I’m sharing my meal with.

100% agree. This article should just be "rewrite your script in Python". It's not the best scripting language ever, but it works fairly well and it's about 100 times better than Bash.

Reply on news.ycombinator.com