- If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python.
- If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date.
Python seems to handle complexity better, but only on the surface. (Basically Python has nice data structures, great string formatting, and .. that's it.)
Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current directory with modules, requirement for python-dev headers, and GCC to compile native parts of packages, and then of course maybe extra shared libraries too) that can affect the "script".
Yet Python provides no way to sanity check itself. Mypy is wonderful, but doesn't really help with scripts, where the enemy is raised exceptions, not simple type errors.
And Python lacks efficient, quick and dirty process control. Sure, there are nice packages for handling subprocesses, but that's really a lot more batteries-included in Bash.
Do I like Bash? No, but debugging a bash script is -x, even the most vile many thousand lines of Bash will eventually stop somewhere with a clean-ish error or finish running.
I have no great recommendation for alternative (maybe Ammonite, maybe scripting in Rust), but I'm interested in what people think about this.
>Python lacks efficient, quick and dirty process control.
Yeah, quick and dirty, that's kind of the problem really. It's great for 20-line scripts but it turns into a huge mess for larger projects unless you enforce super strict guidelines and everybody has their doctorate in bash.
Python, perl and friends make it harder to spawn 3rd party programs and pipe them together for instance, but it also makes it massively easier to handle errors and give decent diagnostics to the user.
I don't have more faith in Python's except than in Bash's trap.
If you use 3rd party code or a subprocess, then both are pretty weak compared to something kernel enforced.
> makes it massively easier to handle errors and give decent diagnostics to the user.
I don't really find that. You have to fight for input/output for subprocesses in Python.
Sure, logging is easier in Python/perl/PHP/whatever, than in Bash, but error handling is about the same. You have to do everything manually, and there's no compiler, nor type (or other magical) system, that'd warn you if you have missed something.
You are aware that error handling is done via exceptions python?
I'm talking about how weak guarantees Python gives compared to Bash about the correctness of the program. It's easy to get None-s, it's too easy to forget to handle file not found or lack of permission.
I've almost never wanted to have my program do anything but crash upon file not found or permission error.
Same goes for permissions issues and a whole slew of similar scenarios. For many use cases, the lack of forced error checking is a feature.
Concrete example: build scripts. As long as the cause of the build failure is clear from the uncaught exception, there's no reason to handle it. I don't need a friendly UI to avoid scaring off non technical users, I just need something that handles the happy path well and crashes with a meaningful stack trace when something unexpected happens.
Trap is not like an exception, it's more like "atexit" or a signal handler. You can't really nest them, you can't really recover from them. So I definitely disagree that error handling is about the same, python's a lot better.
>there's no compiler, nor type (or other magical) system, that'd warn you if you have missed something.
Well that's the whole static vs. dynamic debate but that's a different story. Use C# or Rust or whatever you feel like if you really can't stand python or scheme. Maybe there's a niche to be filled for statically typed scripting languages, there are no popular ones that I'm aware of.
Still, that's besides the point, while I personally prefer static typing like you there's really no contest between dynamic typing and "everything is a string and you can only concatenate them" typing.
Here's an example that builds a React Native app, while updating version numbers and Git tags:
https://github.com/laurent22/joplin/blob/master/Tools/releas...
I love Node but I use it for backend where the event-loop model is useful.
Very useful when debugging.
It's a complete scripting language, with tons of features, from a comprehensive standard library with something for most needs, to a packaging system, isolated environments, async io, and more. And lots of expressivity in structuring your program (including optional types).
So, not sure what the above "and that's it" means. That it's not Haskell? Yes, it's not.
>Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current directory with modules, requirement for python-dev headers, and GCC to compile native parts of packages, and then of course maybe extra shared libraries too) that can affect the "script".
If you're doing dev-ops with such scripts, you already need to handle the environment in general.
That scripting in Python is pretty bad. It makes thing harder and at the same time you still have to think a lot about what can go wrong, because there's no enforced error handling.
> "Scripting" in a compiled language is not practical.
The premise is that scripts grow, and you should switch to Python. Which might be okay for Google, but seems to be only a very small step up from Bash, if any at all. (I've outlined the reasons, why I think it's not worth it.)
I do a lot of scripting, in Bash, and when that stars to grow on me, it's time to think about the problem for real. And usually a blind rewrite in Python is not the way to go.
> For some tasks that's perfectly sensible.
Those are the one-off scripts, no? Anything that should run with some regularity, anything that should be reproducible, that handles valuable data, is not something that should be cobbled together in bash/python and thrown into /etc/cron.d . Or apparently that's how Google rolls.
Yeah, well, maybe not everyone is an SRE, so the use case here seems to be build scripts and other not-so-critical scripts, that still have to be maintained, where knowledge of Bash/POSIX-sh is not really assumed, it should just work for anyone who checks out the repo.
The major disadvantage of any "comfortable" language aside from Python is that they aren't installed by default on nearly every distribution of every OS.
This is why, for example, Ansible only requires Python 2.6 on managed nodes, allowing literally zero installation on almost any host you want to manage.
If you already have a good mechanism for packaging and distributing software to all your machines (which I think is a reasonable expectation for Google) then go ahead and use whatever language you're well equipped for. But know you'll be sacrificing the easy and wide distribution that using Bash or Python brings you.
Yes, Ansible transfers everything. It's ugly and slow :(
I pretty much gave up on [vanilla] OpenStack because their idea of DevOps is custom python script generated Ansible plays.
What it begs is for a new tool. A single file, statically linked “super shell” that lets you do structured scripting and has builtins for a bunch of common things (reading in addresses, daemonizing processes, reading the process table, etc. and that tool needs to be build able on nearly everything. Now I like rust and the attention it is getting and the interest in safety and correctness. I can’t think of many places where I’d reach for bash but be happier writing rust, a lot of it is 15minute disposable programming jobs. At a glance, I think I’d trade 5minutes of set -e to find some error I skipped for 45 minutes of trying to rejigger things to properly borrow some shared value.
But now, to think of it, just using Node + TS (+ yarn) sounds better than Python.
A modern take on Tcl is what we need, in my opinion.
Surely we should be able to build something nicer 30 years after the first release of Tcl.
Has something changed suddenly in the last 5 years? Yes, the docs had a decent API reference but everything slightly higher up than "what parameters this command has" was on that wiki with comments from 2004 marked as out of date by out of date comments from 2007...
Tcl's string representation for objects is just a serialization/deserialization mechanism, which seems to be pretty popular in other languages as well.
Additionally all of the cool things in Tcl such as coroutines, threads (as an additional, but included, package), great documentation delivered as man pages, virtual filesystem access, virtual time, safe interpreters for running untrusted code, a stable ABI (stubs) so that compiled extensions can be used for decades, a small memory as well as disk footprint, extremely cross-platform, easy to embed into larger programs as an extension language, easy to compile a large static program with Tcl as the "main" entrypoint, native BigNum support, many thousands of great packages, ....
What would a modern take on Tcl improve on Tcl that Tcl couldn't just build easier ?
Interested! Available?
[1] https://chiselapp.com/user/rkeene/repository/tuapi/doc/trunk...
It's obviously not perfect, but it has replaced lots of Bash scripts for me.
But probably for deploy scripts, go with a static binary (hosted on, let's say your GitLab instance - using the GitLab HTTP API with a token to wget/curl it) is nigh unbeatable in end-to-end development and deployment time.
So basically, for them, Python requires no fiddling and works as is.
Same for the shell. They say "use [[]]" because they know that compat is not a problem: it's the same bash everywhere.
Google has millions of servers, billions of lines of code and surely of lot of experience with deployment, managing complexity and handling teams.
I'm kinda trusting them on this choice.
Oh, and of course in doing so, I always try to minimize the amount of calling out to other processes. Why call awk when you could use `while read` instead?
You only get one array to work with in portable Bourne shell, $@. You can parse lines using read, and strip prefixes and suffixes using parameter expansion (`${parameter%pattern}`). It's a restrictive little language, with a lot of quirks and edge cases, but it always makes a nice little challenge to figure out how you can use it safely.
One of the constraints I impose is that spaces in pathnames should always be handled correctly; the biggest form of trouble you can easily get in is to forget to quote something or to parse something on space delimiters when it can include spaces. I've seen someone write a script that accidentally turned into an `rm -rf /path/to/important/data` because of poor handling of spaces.
I generally do not attempt to handle newlines in pathnames. While it's possible to do so in some cases with null delimiters, I have never seen a case of newlines in pathnames in the wild. Of course, this means that I have to make sure that these scripts are never exposed to anything that could contain malicious filenames.
Also, I move away from portable shell if it will really make the code too convoluted or fragile. For instance, if I need to parse long options (--foo), I use GNU getopt rather than the built-in getopts (with appropriate test to make sure the getopt I'm using is GNU, so it won't break silently on a system without GNU getopt), or move to a real language.
Anyhow, while I know it's counterproductive, the puzzle-solving part of me really likes trying to solve the problem of implementing something in portable Bourne shell.
I tend to do the exact opposite. If a filename has spaces in it, i like my script to fail spectacularly while insulting the user (who is typically me....)
Filenames are variable names. It makes no sense to allow spaces in them.
#! /usr/bin/env rdmd
import std.stdio;
void main() { writeln(2); }
#!/usr/bin/tcc -run
#include <stdio.h>
int main() {
printf("Hello, tcc\n");
return 0;
}I think it's nicer.
#!/bin/sh
crun() {
local file="$1"
shift
local exepath="$(mktemp)"
if [[ "$file" =~ \.c$ ]]; then
gcc -g -Wall "$file" -o "$exepath" || return $?
else
echo "no filetype detected"
return 126
fi
"$exepath" "$@" & fg
}
#!/usr/bin/env zsh
function crun {
zparseopts -E -D -- -gcc::=use_gcc \
c:=custom_compiler \
o+:=copts \
Wl+:=lopts \
-dump::=dump_asm \
v::=verbose \
h::=usage \
g::=debug
if [[ -n $usage ]]; then
cat <<- EOF
usage: crun [options]... <filename>
--clang (default) use clang for C & C++ files
--gcc use GCC for C & C++ files
--dump dump assembly of program
-o supply an option (e.g -o -Wall)
-v verbose
-g debug
Compiles and runs a C, C++ or x86 Assembly file.
EOF
return 126
fi
# select unique entries of `copts` and then slice copts[2..] (copts[1]
# contains the flag, e.g "-o")
local file=${@[-1]}
local options=${${(u)copts}[2,-1]}
local exepath="$(mktemp)"
if [[ $file =~ \.(cc|cpp|cxx)$ ]]; then
local compiler="clang++"
$compiler -std=c++1z -g -Wall -Weffc++ ${=options} $file -o $exepath
elif [[ $file =~ \.c$ ]]; then
local compiler="clang"
[[ -n $use_gcc ]] && ccompiler="gcc"
$compiler -g -Wall ${=options} $file -o $exepath
elif [[ $file =~ \.(s|asm)$ ]]; then
local objpath="$(mktemp)"
nasm -felf64 $file -o $objpath && ld $objpath -o $exepath
else
echo "no filetype detected"
return 126
fi || return $?
if [[ -n $dump_asm ]]; then
objdump -S -M intel -d $exepath
else
[[ -n $verbose ]] && echo "exepath: $exepath"
if [[ -n $debug ]]; then
gdb --args "$exepath" "$@"
else
"$exepath" "$@" & fg
fi
fi
}I've told developers that if I catch them writing shell scripts in python they must commit a shell script doing the same set of operations. 2x the work usually dissuades from this type of nonsense.
(If you told me I had to rewrite a working Python script in shell, I'd probably think you were joking. Talk about nonsense.)
My route is to make a good rule about writing python with more than 'n' os., subprocess. and *.Popen calls automatically requiring shell script counterparts in case we find your need to pythonize unsafe, unreadable and slow|broken|buggy.
Having a good understanding of shell code will also allow you to easily write slick one-liners that will save you lots of time.
https://www.youtube.com/watch?v=Sg4U4r_AgJU
I'll admit, I'm no fan of Python, but this has little to do with the language itself. I found Bash and Ruby first, so Python just seems like too much work to get right.
Requisite xkcd: https://xkcd.com/1987/
Also, it really isn't 10-20 lines to spit out the contents of some files in Python:
>>> filelist = ["tmp.pl", "tmp.go", "Tmp.hs"]
>>> for f in filelist:
... print(open(f).read())
For instance, if one of my files has a space in it, shell becomes a conceptual minefield, requiring you to understand the half-a-dozen ways it might process strings and how that interacts with filenames and arguments passed to commands, whereas the Python just keeps doing what you probably meant, because it knows what's a string and what's something else. Shell makes a lot of sacrifices for that concision, and there's a lot of "Well, this will probably be OK on anything I run it on...". I never put spaces in my file name, but use periods or underscores instead, precisely so I don't screw myself over with shell. I shouldn't have to do that. Personally I think the crossover point is in the high single digit number of bash lines. The only advantage bash still holds over Python at that point is large pipelines, and 80% of those can be replaced by Python functions. For the remainder, use a pipelining library.
for i in ‘ls /tmp/foo’; do echo $i | grep bar | cut -d’-‘ -f3,4; done
Your code also breaks if the filenames have spaces:
$ ls -1
bar-def-gh i-jkl-mno-p
bar-def-ghi-jkl-mno-p
$ for i in `ls`; do echo $i | grep bar | cut -d- -f3,4; done
gh
ghi-jkl
(HN is forcing me to say [asterisk]. AFAIK there's still no escaping for them.)
You lucked out a bit here too; I don't think I can get echo to do anything very interesting based on file names. Had you interpolated that somewhere more interesting I'd give you some security issues too, if anybody not trusted can create files, which, again, the naive Python code would be much more robust to, because in Python, you have to explicitly ask to bring a string up to an execution context or a function parameter context, unlike shell's complicated rules for interleaving them. You pay a steep price for the convenience.
Of course, this is a matter of scale. I use that sort of pipeline interactively every day. I just start shuddering when it goes in a file for a script, and shudder more if it goes into source control.
On the plus side, when I fixed the typographical issues, it did work. I'm pretty sure I could do this more-or-less first-time right in Python, too, but it clearly took me longer to get there.
And then the parent even specifically pointed out that files with spaces would be a minefield.
Also complaining about looking up docs is just a matter of where you have your knowledge. I would have the opposite problem to understand the "idiosyncrasies" of what the -d flag into cut does whereas I know the python standard library by heart.
This is a cop out. File names can totally contain spaces, and the difference between the Bash and Python solution is that one will continue to work when a file with spaces inevitably shows up and one will fail, possibly silently.
As for file names not having spaces in them, after I said I personally don't use spaces in my file names,
find -name '* *' | wc -l
(Also, because I know someone's gonna try to go for the zinger here, I am not being hypocritical. I do not have an objection to one-off scripting jobs, especially ones like this that are read-only so there is no permanent consequence for error.)
Also with something like Gnu Parallel you can easily scale across multiple machines. Maybe there are more efficient approaches but a pipeline and gnu parallel is pretty easy to scale up without thinking much about it.
I love the way of writing scripts. You start with a simple command and keep piping and saving the output to other functions and commands until you reach the desired outcome. While doing so, you can simply wrap every command into wrapper functions and have the entire arsenal of cli applications at you disposal. While not having to care about such esoteric/complicated/efficient stuff like data types as everything is a string until you have a command which knows how to interpret it otherwise.
But at the same time those scripts tend to be very slow, lavish about resources (e.g. compared to compiled languages C/Go/Rust) and the practical/secure syntax is just awful. When simply saving the output of a command to a variable uses six special characters, you know you are coding (s)hell:
a="$(echo fun)"
Despite the rough syntax I really enjoy writing bash scripts. Maybe its just that I am able to solve everyday problems with just a very few lines of code.
PowerShell is materially worse than bash for a great many purposes because of this design choice alone.
[0] https://brianreiter.org/2010/01/29/powershells-object-pipeli...
Historically, PowerShell was supposed to be a replacement for cmd.exe and roughly equivalent to bash, yet suitable for the Windows environment. They tried early on to port some of the UNIX & Linux tools to Windows and it didn't fly, so this is what we're left with.
Although parts of Powershell are open source now, I suspect the mere existence of WSL means that Microsoft understands it has more to gain from compatibility with the rest of the world (Bash is nigh-everywhere) than imposing their particular philosophy on people.
(See also, Apple's decision to adopt UNIX underpinnings)
I'm prepared to be wrong if PowerShell actually gets used outside of Windows, but I wouldn't bet on it.
In PowerShell you pass structured data around instead of globs of text, which is much more maintainable. Using Bash on Windows is like going backwards, and it's sad that most developers don't even realize it.
> Historically, PowerShell was supposed to be a replacement for cmd.exe and roughly equivalent to bash, yet suitable for the Windows environment. They tried early on to port some of the UNIX & Linux tools to Windows and it didn't fly, so this is what we're left with.
Nonsense, Powershell is a fundamental improvement. It's not a "this is what's we're left with" situation.
> Although parts of Powershell are open source now, I suspect the mere existence of WSL means that Microsoft understands it has more to gain from compatibility with the rest of the world (Bash is nigh-everywhere) than imposing their particular philosophy on people.
Yeah, let's just drag everything down to the lowest common denominator.
> (See also, Apple's decision to adopt UNIX underpinnings)
Which is sad, since it could have been so much more. The Unix way of doing things, isn't the be all and all.
There are advantages to having a pipe based on structured data, but that also means you have to know what structure to expect at ever step of the way and whether or not other tools can work with that structure. There are also tons of little quirks about Powershell; enough so that I eventually just started writing most stuff in C# and then just using the Powershell scripts as glue.
Passing text means that every single one of the thousands and thousands of unix cli tools that accept stdio will work in your pipeline.
I think this industry needs to get away from the terrible/horrible, absolute thinking. Most tools have positive and negative
It also means every one of those thousands and thousands of Unix clip tools needs to have a parser to turn that text into some sort of structure to operate on.
It'll be a long time before the Powershell ecosystem has a fraction of tools that are compatible with it's .NET object pipeline.
Up until someone changes the text output of the program and it all goes bad. Never mind the maintenance headaches that basically get solved by people moving their command lines into databases where typing happens.
It'll be a long time before the Powershell ecosystem has a fraction of tools that are compatible with it's .NET object pipeline.
That's more a function of age and developers than the typing itself.
It would be, if cmd.exe was long dead, PowerShell had stolen the good parts and continued on its way, but we're not there yet. Cmd.exe still exists on Win 10 and let's not even get into the fact that 64 bit only versions of Windows are not yet everywhere. Sure, Microsoft's ecosystem is larger and more complex, but at some point Win32 becomes a liability and unnecessary baggage.
> Which is sad, since it could have been so much more. The Unix way of doing things, isn't the be all and all.
Cultural compatibility matters - See C & C++. How else do you expect to get users and developers on board if you can't show them how your system is better in a few key ways, but isn't a giant leap from what they're currently using?
Tl;dr Change is scary, try to make it less so.
As a 20 year bash veteran: powershell is the first OS default shell that outputs structured data, and looking at objects and selecting the properties you want is a massive improvement than combining bash with sed/grep/awk to scrape text.
Someone bizarrely responds that cmd still exists on Windows for compatibility purposes (though even Win+X starts powershell now) doesn't change this at all.
The README for my powershell profile (which is written from a *nix PoV) has a little more info comparing the two: https://github.com/mikemaccana/powershell-profile
How is it that objects are an accepted thing for basically every programming environment in modern use, yet somehow controversial when it comes to the shell?
The prayer-based text parsing toolchain sucks. It has always sucked, regardless of platform. We put up with it because it was all that we had.
Jeffrey Snover came up with something better and thanks to PS Core we'll be able to use it everywhere!
They aren't accepted for basically every programming environmental nment in modern use; peak OOP-all-the-things is well in the past, and it's no longer the one paradigm to rule them all, irrespective of use case.
What? PowerShell has radically different ideology. Plus it is super consistent (helps discover commands), extensible and works with objects rather plain strings, use .NET functions or call custom .dll functions...
> They tried early on to port some of the UNIX & Linux tools to Windows and it didn't fly
Actually they just use aliases to map well-known unix commands to powershell counterparts. It is not expected to have command switches work like that etc. Just a convenience for folks used to rm, ls, tee, wget, whatever. Maybe bad decision because of confusion, maybe a good one because helps people get started with PowerShell.
For me as a Windows admin, PowerShell is one of the best things within Windows Ecosystem.
Basically, text files (UNIX) v structured data from API's (Windows), thus PowerShell was born.
https://www.heavybit.com/library/podcasts/to-be-continuous/e...
You might know that if you'd spent more than 30 seconds with it before shouting "it's not bash!".