Ask HN: Let's build Checkstyle for Bash? | Hacker News Reader

Ask HN: Let's build Checkstyle for Bash?

59 pointsa-t-069 comments

After working with Bash and Shellcheck for a few months, I noticed I could improve my code quality by making it compliant with the Shell Style Guide by Google [0]. While working on that, I thought some aspects of this Shell style guide can be verified automatically, granted some assumptions/opinions are formed. So I looked around for linting tools and autoformatters for Bash:

Shellcheck: https://github.com/koalaman/shellcheck

From Asynchronous Lint Engine (ALE): https://github.com/dense-analysis/ale/blob/master/supported-...

- bashate: https://github.com/openstack/bashate

- cspell: https://github.com/streetsidesoftware/cspell/tree/main/packa...

- Bash Language Server: https://github.com/bash-lsp/bash-language-server

- shell -n flag: https://www.gnu.org/software/bash/manual/bash.html#index-set

- sh(shfmt): https://github.com/mvdan/sh

- shdoc: https://github.com/reconquest/shdoc

From this stack post [1]:

- checkbashisms: http://man.he.net/man1/checkbashisms

- shlint: https://github.com/duggan/shlint (archived)

Prettier: https://marketplace.visualstudio.com/items?itemName=esbenp.p...

Within all these linters and auto-formatters I did not find checks that enforce, for example, the Function Comments of the Shell Style Guide by Google:

All function comments should describe the intended API behaviour using:

    Description of the function.
    Globals: List of global variables used and modified.
    Arguments: Arguments taken.
    Outputs: Output to STDOUT or STDERR.
    Returns: Returned values other than the default exit status of the last command run.

Hence, I thought we could make a Bash linting tool that verifies compliance with the Shell Style Guide by Google. To do so, a brief start was made here [2]. It identifies/lists elements in that style guide that may be verified automatically. Since Bash has been around longer than me, I think there may be some people better suited for the development of this enhanced linter. Hence, I thought it might be wise, for impact and usability, to share this idea here.

What do you say, HN?

[0]: https://google.github.io/styleguide/shellguide.html

[1]: https://stackoverflow.com/questions/3668665/is-there-a-stati...

[2]: https://github.com/TruCol/checkstyle-for-bash

I've always been confused that in ${current_year} people are still directly writing huge + complex shell scripts.

Compare/contrast Javascript, another language we're "stuck with": while you can still directly write it, many people don't, instead using transpilers to target Javascript as an "object code" from some other, stricter language (e.g. TypeScript, ClojureScript, arbitrary native languages via Emscripten / WASM compiler targets, etc.) Via these languages/compilers, everybody who wants compile-time strictness can have it. And you can even get runtime strictness that Javascript itself doesn't have, as the checked semantics of these languages will often be "lowered" into the resulting Javascript by generating explicit assertions (usually this is on by default, unless you ask for an optimized build.) But the output is still just "Javascript", that any browser can run.

Why hasn't a similar thing happened for POSIX-standard Bourne shell? Why are we seemingly "satisfied" with writing + maintaining 5000-line install.sh scripts in our codebases, with a bunch of extra mental overhead / tooling required to keep the code sane and clean; when we could just be writing in a sane and clean language to begin with?

Is it just that nobody's bothered to create such a language/compiler? Do I need to be the one to step up, here?

I don't think I understand. Javascript is required in the browser as the end target, but the operating system exposes the APIs which are consumed by bash to do useful work. The normal solution to not wanting to use bash is to... use another language (including, ironically, javascript).

Much of the time that people are writing shell scripts, they're writing them not because they prefer shell syntax to that of some other language, but rather because they're creating a script that needs to be widely disseminated/deployed to all sorts of machines with unpredictable toolchain availability.

This is why a large fraction of the shell scripts that exist in the world still hold to Bourne shell syntax, rather than using any of the syntax extensions from its descendant shells: Bourne shell (or at least, something at /bin/sh that interprets Bourne-shell syntax) is part of the POSIX standard. So you can expect any POSIX system — no matter how weird — to be able to run (Bourne) shell scripts. You can run them on Alpine. You can run them on Busybox. You can run them on your NAS. You can run them on your router. You can run them in your initramfs, on your Kubernetes nodes, on your Mosix cluster, on your smart TV, on your watch, whatever. They run on Windows; they run on macOS; they (obviously) run on Linux and BSD. New Bourne shell scripts written today, run on 40-year-old operating systems nobody even remembers.

(If you think about it, the whole way that GNU autotools does its job, is by relying on the portability of POSIX-compliant Bourne shell syntax — i.e. the ./configure script — to bootstrap out enough of an understanding of whatever system it lands on, to find the tools required to then probe the system for the availability + capabilities of its compilation toolchain.)

For a small example of the type of script I'm talking about — see e.g. the script you download+run when you run the command-line on https://rustup.rs: https://github.com/hsivonen/rustup.rs/blob/master/rustup-ini....

For a much better example (that I sadly can't link to), see the install.sh for VMWare Workstation for Linux.

There's absolutely no benefit that these scripts get from being developed directly in POSIX-compiliant Bourne shell syntax, rather than being developed in something that compiles to said syntax; any more than programs for your PC would benefit from being developed directly in ASM, rather than in something that compiles to it.

The niche for such a language would be extremely narrow compared to existing better-than-shell-script alternatives. Wasn't that perl's selling point? Compiling to bash doesn't seem like that much of a better option than shipping (or just requiring implicitly) a $BETTER_LANGUAGE runtime, and writing in that language. (I imagine someone might respond with some carefully constructed scenario where you can only use a shell script, but I'd expect that sort of scenario to be rare.)

My go-to is either Python or Ruby when shell scripts start to get unwieldy - lately it's been Ruby but I'm mentioning Python because I have experience with the type-checking options for Python, and you mentioned that as a benefit. I'm sure someone will point out that type-checking exists for Ruby too (Sorbet, others?); I've just never tried it.

> I imagine someone might respond with some carefully constructed scenario where you can only use a shell script, but I'd expect that sort of scenario to be rare.

Not to comment on how frequent this is for other people, but my rare, obscure, and carefully constructed scenario is this one:

    docker run --rm -it ubuntu

I don’t think many are happy writing that much bash. I certainly wouldn’t be! I think this problem is completely solved by other languages though.

Bash is popular because it’s available (nearly) everywhere, is very portable, and has a massive “standard library.” (Technically, not a standard library, but you can basically guarantee that things like sed, curl, rsync etc. will be available or easy to get anywhere.) And its constructs are very well fitted to smaller scripts and automations.

You loose those benefits with a compilation step. Nearly all the scripts I interact with are just smaller automations put into a “bin” folder in the repo. These scripts are used for the build steps of other programs. So what would you write for automations related to building a compiled bash project? Probably bash lol

If you’re writing thousands of lines, you can just use any other language, right? Pretty much any other scripting language can execute commands in bash anyways. To my knowledge, stepping into a different language once you need more refined flow control and data structures is pretty normal. Afaik many languages can either produce output which can be run directly on the CLI (rust, C++, C), and others can be executed if you have the language runtime installed (JS via node, python, etc.)

> Nearly all the scripts I interact with are just smaller automations put into a “bin” folder in the repo.

What you're talking about are scripts you (or developers you work closely with) write to then use yourselves, in development workflows. These tend to be small and simple, and also places where the pain-points of shell-scripting (e.g. spaces screwing up path parsing) don't tend to come up, because developers have enough sense to not make their development environment actively hostile in these ways.

But this is not the only, or even the main, use-case of writing Bourne shell scripts. (If you have a development toolchain installed, you probably already have Perl and/or Python and/or Ruby — so why not script your development workflow in one of those? Or, if everyone is at least running a modern Linux or macOS+Homebrew, why not assume the existence of Bash or Zsh, and write for one of those, instead of Bourne shell?)

Instead, the main places where large, complex shell scripts are necessary, are in:

1. multi-platform software distribution;

2. systems-level glue code (think: initrd scripts; sysvinit back when that was a thing; etc) that must be runnable "at early boot" or in a "rescue environment" — i.e. one that can't guarantee that /usr is mounted, or that /lib is set up correctly to make non-static binaries work, etc. (Or in an "embedded environment" — in Busybox, /usr doesn't even exist!)

In the first case, the person writing the shell script is not the person running the shell script. Tons of different people run the shell script, on (sometimes drastically) different environments, and these are not developers, and their machines mostly aren't guaranteed to have anything like a development toolchain installed.

In the second case, despite targeting a single environment, said environment just happens to have a Bourne-shell compatible interpreter as its only scripting environment. (And probably even that was only included because the environment wanted to claim POSIX compatibility, never expecting people to want to script it rather than compiling native tools statically into the environment. Even though most sysadmins who might need to modify these environments, have no idea how to do that.) Mind you, some of these systems — e.g. UEFI — aren't POSIX-compatible at all, but rather require you to "script" them in something totally different, e.g. Forth.

> If you’re writing thousands of lines, you can just use any other language, right?

No, not if you can't make any assumptions about the target environment beyond "it's POSIX"; or if the target environment really doesn't offer anything not strictly required by POSIX. There is no POSIX-standard scripting environment other than Bourne shell. (And you can't just ship a binary, because you don't know the target architecture. Detecting the architecture you've landed on is probably part of what you're trying to do!)

This has been tried before. One of the results is autoconf.

At the end of the day, it's probably less painful to just bootstrap some saner interfaces if you really need them. The POSIX C API is just as standardized and exposes more than the shell interfaces.

That assumes you're targeting systems with a local compiler toolchain. A valid assumption for autotools itself, but "having a C compiler available" isn't a requirement of POSIX. (POSIX merely standardizes the interface that such a toolchain should present, if one is available.)

I reiterate my example: the install.sh for the Linux edition of VMWare Workstation. Its job is to take an arbitrary machine running a desktop Linux OS (i.e. it does assume an X installation), and turn it into a hypervisor dom0.

To do this, it needs to compile its proprietary Linux kernel modules using DKMS. But 1. it can't assume that you have a compiler toolchain installed; and 2. it can't assume that you have the kernel header package installed for your kernel.

So this install script needs to detect your OS distro-family, and use that to determine how to forcibly install some packages, in order to then have the tools (and headers) available to compile things.

How would you suggest that this installation logic be implemented? Because right now it's "thousands of lines of extremely grody Bourne shell script" — and AFAICT, that's because that was the optimal choice given the problem domain.

I suppose you could build such an installer as a collection of static binaries, one for each architecture. (Static, because different distros name their libs differently, so dynamic linkage isn't portable.) You could then create a little install.sh that runs them.

And, if you wanted all this to be tucked into a single file so you could run it with the unsafe curl-to-shell hack, you could even base64-armor those binaries and drop them into some heredocs inside a bootstrap shell-script, such that you do get to have a single "install.sh", which just pops out copy of the 'correct' binary into existence and runs it.

But IMHO, by doing something like this, you're just Greenspunning a Bourne-shell object-code format; the same one a compiler could emit. But without the benefit of getting to directly use a linker to generate the result.

Sane people don't write shell scripts. I script everything little more complex with Ruby/JS.

Oh people have tried - here are a few https://stackoverflow.com/questions/10239235/are-there-any-l...

I vaguely remember quite liking bish when I saw it years ago https://github.com/tdenniston/bish but it looks like no commits in 6 years.

This shelljs thing looks more promising, but really tedious to use https://github.com/shelljs/shelljs - shell.rm('-rf', 'out/Release'); I'd rather suffer proper bash than have to do that sort of thing.

Nothing seems to have really caught on so far. Bash is easy to learn and hack on, and before you know it, that simple install.sh that started out moving a few files around is 5000 lines, unmaintainable, and critical to bootstrapping your software :)

2 things I'd love to see in a checker would be:

Warn me when I'm using -h instead of --help. When writing persisted scripts it's often more descriptive and self documenting to use long form flags[0] when a tool gives you both options.

Warn me when I mix using "-p 8000:8000" and "-p=8000:8000". I know it's technically not possible to give a definitive answer here because supporting spaces vs equals comes down to the CLI tool itself but you can pick up when you mix the 2 styles and leave it up to the script author to pick a style and stick with it when possible.

I do a lot of shell scripting in my day to day (ops, etc.) and it's always a manual process to give feedback on code reviews when others contribute patches or introduce new scripts.

[0]: https://nickjanetakis.com/blog/when-to-use-long-word-or-shor...

@ nickjj Thank you for the input! I have converted your comment into the first issue: https://github.com/TruCol/checkstyle-for-bash/issues/1

Take a look at shfmt and shdoc. One is an auto formatter, the other does autodocs. I wouldn't invent yet another docstring format. It's already been done. Just check compliance with an existing format.

@bashhacker, thank you for the suggestion! I included shdoc in the list, and added the synonym shfmt to the sh formater (which includes shfmt).

I agree with not making another docstring format. Accordingly, I updated the readme to explicitly also allow the linter to support the docstring format as provided by another style; that of shdoc.

In essence, the user should be able to configure the linter to adhere to Google Shell Style Guide or the one used by shdoc (or another, or some non-conflicting combination of options). To start somewhere, I propose Google Shell Style guide.

I just did it. I use Markdown for bash-modules documentation. :-/

A procedure in bash can have way more complex interface than simple function call. See http://vlisivka.github.io/bash-modules/arguments.html for example.

Out of pure curiosity, in what context do you write sufficient amounts of Bash scripts that style checking is a worry that needs your attention? While I also write small one-offs or bootstrap scripts here and there, in most cases it's my experience that developers opt for other languages for anything beyond small snippets.

Largely same sentiment. Pure POSIX shell scripts for embedded systems, sure, but I'm not sure if there is a niche that requires Bash scripting. Not discouraging these efforts of course but I'm not sure that it'll be worth it.

The niche is installer scripts. Because there is one certainty: Bash is available everywhere.

> Bash is available everywhere

POSIX mistake number 1: Bash isn't available everywhere, even when restricted to Linux. Even Debian avoids bash for a good reason (Bash is slower than most POSIX shell implementations), although it's installed by default for convenience.

> Even [on] Debian ... it's installed by default for convenience.

:-/

zinekeller's point notwithstanding, i agree that "installer scripts" is really the last necessary domain of the bash script. If you need a (semi)portable script to get `python` to run your python script, you can't use python, obviously.

Things like docker images and nix builds are sold as huge improvements on this chicken-egg problem, at least from a "keep it all in one language" perspective, but only if you consider a docker script (mostly bash) or a nix config truly not the same thing as a shell script that more or less initiates the same environment.

docker/nix abstraction layers have advantages but aren't always as portable as a shell script. If you need a docker engine, I think you'll still need a bash script to install that...

Nix is possibly even more "single command", often just a curl if i'm not mistaken, but still requires config and shipping a build somewhere, and you still need to run the `curl` command in a shell somehow. i don't see how "just use python" will ever fix the init chicken-egg problem.

What do you use the; if you want:

- Self contained in one file

- Easily edited with vi/nano (i.e. through a console ssh session)

- Readily available on most Linux base installations

- No crazy runtime installation requirements

Edit: formatting)

python.

perl

Not my cuppa, but I'd take it over bash any day

Which python? 2.7 that is symlinked to `python` on the distro? 3.5 that is installed by default? or perhaps you meant 3.9 but what about that cool library that you're gonna use which is only available for 3.9.2 and not 3.9.1 due to a bug?

How are you gonna handle installing that beautifulsoup dependency in a way that does not bork the OS python deps?

Are you gonna use `python -m venv` or `poetry` or `pipenv`?

If it is for myself: murex (my own DevOps focused shell: https://murex.rocks)

If it is for work: either Bash or whatever that companies preferred software development language (over the last ~20 years that's included: Pascal, Visual Basic, PL/SQL, Perl, PHP, Python, Typescript and others I've likely forgotten. Different companies will have different preferences)

If performance matters then these days Go is my "goto" language because it's a good compromise in terms of developer productivity and application performance. But I've used Java, C and C++ too. Depending on the particular problem I'm trying to solve.

...and if you ask me again in 5 years then the answer will be completely different again.

The sort answer is: there's no such thing as a single answer. Everyone will have their own preferences and business requirements.

Some context to my statements: I'm a SWE that works on infrastructure services and tooling, primarily on Linux but also Unixes.

It depends on what I'm doing but usually Go if an application requires any longevity at all.

I use Python or Java for things that require more dynamic typing. The main usecase here is big data or to prototype applications.

Bash I usually reserve for very small tasks. When I bootstrap services in containers, you'll usually find some Bash. A whole CLI written in Bash? I would not approve that PR. Bash lacks variable scoping for the most part, the ways in which you can implement it are hacky and non-obvious. Bash's syntax and the way it behaves varies by system, even installer scripts have to account for this. As I'm typing this, I realize the Bash code I write generally follows the same testing and constraints as the Makefiles I write.

I don't think I really understand the requirement for "a single document". This departs from programming practices I've built up over the years. When I come across large multi-thousand LOC documents (in any language, much less Bash) I'm usually concerned.

About the requirements and "single document":

Those tasks that I use bash for are usually one-offs, and live on the server they are used on.

Thus, it should be easy for the next admin/Dev to find it and understand/fix/extend it.

Of course, if there is a more advanced task those requirements fall short and there should be a proper Dev env with source control, tests, etc.

Ah, makes total sense. I was conflating the single document requirement with the original post which was seemingly trying to build a CLI tool.

Every web application I work with has this idea of a "run" script[0], basically a self isolated shell script to help automate running certain commands. Sort of like a Makefile except with the full power of your shell.

For example: https://github.com/nickjj/docker-flask-example/blob/main/run

In a bigger project this file tends to grow quite large, even when you have the power of a programming language available to you there's plenty of tasks where it makes sense to keep it as a shell script. Now imagine you work on a team where other folks want to contribute new functionality, it's important to have automated tools to help stick with a consistent style, just like you would want to use Black with Python.

The run script for my infrastructure repo at one place I work for is almost 1,500 lines of shell scripting that's broken up into many small functions. It's not a big jumbled mess either, it's very organized and optimized to make running long commands faster and less error prone (even if they happen to be called in CI). These functions are mostly calling out to Terraform and other tools too, it's not 1,500 lines because of long winded cloud CLI tool API calls or anything like that.

[0]: https://nickjanetakis.com/blog/replacing-make-with-a-shell-s...

I think linters are just as helpful for 10 lines as they are for 1000. I (try to) use style checks and formatters for anything and everything I version-control, especially when there’s more than one person working on it concurrently.

Google's styleguide has incorrect bash being used, they are using string comparison operators for numeric comparisons (see the various `$? != 0` and $PIPESTATUS).

    [[ -1 -eq 0-1 ]] && echo true || echo false
    [[ -1 == 0-1 ]] && echo true || echo false

https://www.gnu.org/software/bash/manual/html_node/Bash-Cond...

The style guide always uses (( so the operators are numerical comparison.

See also where it says: > For preference, don’t use [[ … ]] at all for numeric comparisons, use (( … )) instead.

This is a neat idea, but I'm not sure I see the need for this specific tool. Shellcheck covers essentially all the errors that are mentioned in the Google Style Guide so I'd just be using that instead.

And as the Guide itself, it is mostly OK, but there are a few things I really don't like. For example:

- Whatever doc format they use is not as nice to use as shdoc. Namely, you don't have to add the spaces to your comment to make it formatted properly

- The 'no tabs' thing is pretty bullshit - Bash is literally the perfect language to use tabs and their "Whatever you do, don’t use tabs" comment really kills me

- I don't like their recommendation of using `[[`. Only using `[[` for stuff like regex and glob matching makes it much easier to paste code between Bash and POSIX sh.

There are also a few things they forgot to mention, like the lastpipe` shopt option when talking about piping while loops. And maybe it would be nice to recommend using local like `local var=`. But I guess this guide is meant more for new people that don't know all the pitfalls and stuff

But I think I agree there is some need for a more extensible linter. I've been eyeing the Bash LSP, which is based off the Bash Tree Sitter parser, but that has quite a few bugs with parsing that I've been meaning to send PR's for.

I wonder if `shfmt` has gotten better since I used it. A few years ago, it was quite buggy with prepending extra whitespace whenever I formatted the file

If your problem can be solved neatly with some combination of existing shell utilities, and you just need a bit of glue to join them together, bash is great.

The huge weakness of bash is the data mangling that is more complicated than a relatively simple per-row transformation. Splitting, joining, sorting, min/max of data structures like arrays, lists and trees are trivial in python, not so in bash. If it's not trivially seddable, and awk doesn't help either because you have to rearrange tree like structures, you are about to experience a massive pain, because it's out of scope of even expertly used associative arrays and you will find yourself reinventing the wheel hard using pretty basic tools.

TLDR; As soon as you need to use something more complex than a for loop, switch to Python.

Intellij has a builtin plugin that integrates with ExplainShell, Shfmt, ShellCheck.

I think we should really be working to deprecate large bash scripts. If you write enough to care about code quality you might want to just use Python.

I'd say look at things like Oil(Is that project ever going to be the next big thing like it claims?).... but writing large scripts in ANY shell doesn't seem like the best plan.

What about an Ansible linter? Do those exist? I'm starting to suspect Ansible might be a better choice for a lot of what people do with bash, even if you are running om a desktop.

Shell scripts are nicer than Python, though. There was a brief moment in time where Python nearly rivaled shell scripts in convenience and straightforwardness, but November of 2008 was many years ago, now.

We should probably deprecate Bash, though. It doesn't offer enough over the standard for it to be worth it.

> We should probably deprecate Bash, though. It doesn't offer enough over the standard for it to be worth it.

Arrays make a world of a difference. Deprecate sh, keep bash.

That's a terrible idea. A shell scripting language doesn't need to do everything in the world. It just needs to tape together tools that can do everything.

POSIX sh can pass arrays around just fine, and that's all it needs to do. Feature creep is the worst possible thing you could do to a shell scripting language.

No it can't. You have the argument array which is a horrible workaround. If you need to write scripts that interact with other tools a lot, bash is the language of choice. Starting and handling other processes is a messy thing in any other language.

It's not a terrible workaround; it's a beautiful demonstration of how flexible the standard shell is.

It's a bit of an inefficient hack compared to a language feature.

Which POSIX sh do you have in mind and how does one pass arrays around in it?

POSIX sh. Pass it as a string, use standard programs to turn it into what you want. Voila. Not that hard. Just like you do everything else in UNIX.

If you mean a standard, it will be too abstract to be considered as something that can run anything. Sometimes it has to be too vague in order to not make existing Bourne-style shells non-compliant. This isn't, for example, a standard for the C programming language where you can throw some things into the UB bag or the ID bag. POSIX sh is more what you'd call "guidelines" than actual standard.

Even if it were an implementation, how would you write a "POSIX sh" equivalent of these lines taken from a bash script? (My point isn't that it's impossible.)

mkdir -p /tmp/nodes/{a,b,c,d,e} ; NODES=( /tmp/nodes/{a,b,c,d,e} ) ; echo ${NODES[$((`date +%-j` % ${#NODES[@]}))]}

But POSIX sh doesn't have array support?

99% of my use case for Bash over POSIX sh is arrays, associative arrays, BASH_SOURCE, and PIPELINE (or whatever that array is that has positional pipe return status)

This. The problem most people have with bash scripts is that they are mostly writing sh scripts and occasionally throw in [[ ]] but still use 99% of the ugly workarounds and tricks you had to use in sh to get anywhere.

> I think we should really be working to deprecate large bash scripts.

I second this: I myself have a problem here, which, I guess is somewhat related: I try to keep everything universal and each time I try to generalize what I'm trying to achieve through some sort of abstraction. As a consequence I keep adding functions and aliases to my (zsh/bash)rc to the point where they become absolutely unmanageable. At this very moment zshrc is 8000+ lines long. Sure, I have a cronjob which backs it up occasionally but it's more of a maintenance/readability hell.

Have you considered that your run config is probably not the place for this sort of thing? You could instead have focused libraries in smaller files located at /usr/local/lib/ or wherever tickles your fancy. Then source them as needed from scripts.

I do that too but usually ends up being harder to keep track of them and backup, even if I keep them in a separate directory and added it to $PATH.

Wow, 8 thousand lines is pretty insane. When my bashrc started to get long, I started to separate things into different files and source them. And then that got too complicated so then I made them separate executables with different directories. But then I couldn't reuse functionality as easily so then I found myself making a package manager. And now I'm moving some of that stuff _BACK_ into my bashrc (sort of). It seems there is really no end...

This is one of the reasons I like GUIs. I can recreate almost all my configuration from scratch on a brand new Mint install in 15 minutes aside from a few VS code settings I'd need Google for.

My .xonshrc just has a few visual tweaks. I have set up exactly zero cron jobs aside from maybe whatever mint's backup stuff uses for a timer for the snapshots. I don't have any custom bash scripts that I use, other than some build scripts and things like that, that live in project repos.

I call it "decustomizing" as a counterpoint to minimalism. I don't try to simplify my life, I try to remove customization and unusual tech, even if that means using a 100MB app instead of a 1kb script I'd have to maintain myself.

I've just... had about enough of small time garage tinkering level projects at this point, and have really started to appreciate monolithic opinionated systems. Stuff that lets you do anything, in exactly one way.

Where do you draw the line as "large"?

I just threw out a few hundred lines of Python in exchange for ten lines of bash script + cron job to automate nightly app updates. Way easier to get working, way less painful to debug.

I grant, however, that it's not particularly large. What is?

What Python is to Java, Bash is to Python.

Many simple file and data munging tasks can be done on cli 100x faster with 10x less code than Python, using rudimentary bash-fu.

I already see a wave of Python undoing in the data world. Python becoming the cobol of this field.

I just learned about rev; rev plus cut could have saved me years of writing one off Python scripts to cut out the exact variable I need, haha

rudimentary bash-fu

I wish there were some guides on how to get past rudimentary bash-fu. My bash text munging is still pretty rudimentary. I'd love to get to journeyman text smash status.

You might want to check out 'hck' to replace 'cut'.

https://github.com/sstadick/hck

I may give it a try - I have a bias to stuff that comes preinstalled, though, as I know I'll always have the chance to use it on a stock linux box.

So your 10 lines of bash use a bunch of other programs to do almost all the work, right?

So what you need is a good language that lets you concisely express ways of calling out to other programs. Not necessarily a language that's backward-compatible with whatever seemed like a good idea for a CLI in 1979.

Not even a bunch. It's calling a scraper written in Python, date to add some timestamps, git to add new db entries and push to version control, and touch to reload the webapp.

All that in... (goes off to wc -l the script) ...twenty lines of bash, a third of which are comments.

> language that lets you concisely express ways of calling out to other programs

Seems like bash meets those requirements to me, dawg. /shrug

... but you cut off the word "good"... :-)

"Good enough" is good in my book, and bash is good enough for me.

Still wish someone would give me an idea of what "large" means in bash scripts.

Can you recommend any modern glue languages with the following attributes?

- concise (another operator instead of '|' is fine, but no extra parentheses, quotes etc.)

- performant (at least as fast as Bash)

- standalone (extensions discouraged)

When you start to need error handling.

...unix return codes?

Sorry even I can't say that with a straight face XD

Ansible has a linter. We do use it on our playbooks (locally and in CI). There's also a framework for testing and linting ansible plugins (ansible-test IIRC), though that's been a little painful for us to setup.

That said, we do have a lot of helper scripts that we use in our day-to-day (local development). They're not "large bash scripts", but they're small enough that we don't really need a proper programming language. Some degree of code quality with style guides and shellcheck does help.

Seconded. I use Python daily and bash monthly at best. After the Nth time looking up how to do specific kind of conditional check in bash, I gave up started writing all the extra scripts and tooling I need using Python instead.

Reply on news.ycombinator.com