[ -z $var ] works unreasonably well
vidarholen.net
vidarholen.net
At one point in history, options were more limited. Now, it is just not worth the time to read code that may or may not have obscure errors in its very expressions that you must understand before you can even begin to examine the real purpose of the program!
And once you do, shell is really, really useful and convenient.
For example, Linux distributions are full of shell scripts that would be frankly awful to write in any other language that I know of... like this, to take a random example:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/applicatio...
I still disagree with the ideology that any non-trivial script needs to be rewritten in Python. The shell is the standard way of interacting with a *nix system, and for me that's the major benefit of using shell for "system"-ish tasks. I already know how to make a symlink, a pipeline, a chmod, etc, so I don't need to learn a new syntax for that.
The semantics of shell string interpolation are generally not that hard to learn, compared to learning all the nuances of programming in general that can make your program incorrect, so I find it a bit strange how it's considered to make shell completely useless for scripting.
Shell scripting has been working pretty well since, like 1975! It's not that bad!
Which works as long as attacker (or sleepy admin) doesn't happen to create suitably "bad" path
> Shell scripting has been working pretty well since, like 1975! It's not that bad!
Considering the amount of borked system because spaces/uninitialized variables -stories I've heard over years, I'm not so sure about how well it has been working...
This is much nicer than I expected, but the functions are scattered around a bit.
Alternatively there's always the option of using something like sh[0] and just shelling out to the existing utilities, if you don't mind the portability loss.
'f{out}/libexec/docker/docker-proxy'
f'{out}/libexec/docker/docker-proxy'
Not really, I've also forgotten to convert a few dashes to underscores in f-strings and probably made a few other mistakes along the way. It doesn't detract from the main point: it's a rough estimate of how such a converted script would look.
Though it makes me wonder if PyLint already has a lint for strings which start with F and have f-string placeholders.
>> once you do, shell is really, really useful and convenient
Part of the issue here is that a large number of people who think they know how to write safe shell scripts do not in fact have the ability to do so.
There are so many gotchas to learn that it's all too easy to believe oneself competent, while being unaware of just how short one's knowledge falls. It's just too easy to write a script in the shell of your choice that works - until it encounters an edge case scenario that wasn't planned for, at which point the results can be catastrophic. This is true of any programming language, but the tasks for which people write shell scripts (such as massive system-wide filesystem manipulation) makes the consequences of a single-line bug very real.
Glad you mentioned that, because that was going to be my reply after reading the first half of your post.
I'm not sure a person who is fairly well-versed in shell scripting is going to do a worse job than trying to use python, perl, lua, etc. if he doesn't also hsve a high level of experience and compentence in those languages.
It has been literally 10 years since I touched Python. I use shell scripting every day. I know which one I'm going to be better at using to produce reliable code that does what I want it to do.
That's fine. Python 2.6 is still in wide use.
For my day-to-day shell, I use fish. IMO, it has much better defaults than bash and zsh, and it's fast. I still have to use bash sometimes, but it's becoming increasingly rare.
Your linked example doesn't really seem particularly tricky to me. I'd feel comfortable writing it in something like node or ruby, and I believe the result would probably be easier to consume for certain people.
I just don't think we should reject shell scripts entirely because some care is required to write them correctly. Shell is way too valuable.
I don't reject shell scripts, though. I agree that in some cases it can lead to much simpler code. The power and composability with pipes and immediate substitutions is incredibly powerful. I've had a few cases where I replaced far more complicated node scripts with simpler shell scripts.
What I'd truly love is a reference guide that showed how to correctly do certain things with shell scripts. Here's a real-world example I struggled with: Writing end-to-end tests with selenium and a test runner. I wanted a script to start up a selenium server and wait for it to be ready. Then it should start the test runner. If the test runner exited successfully, kill the selenium server and browser, and exit with a 0. If the test runner or selenium server crashed or failed somehow, clean up and exit with a 1. Even after much trial and error, I still ended up with occasional Chrome copies staying open after failures.
That kind of reference guide would be great. I haven't checked out any of the books on shell programming, but there should be a decent audience for a website with just good patterns for shell scripts, like you describe.
Trapping signals and keeping track of child PIDs is quite tricky, yeah. (It's quite tricky in any language!)
You might be interested in my "wd" program that's basically a bash wrapper for the WebDriver API, which I made because I wanted to write end-to-end tests and automation tools in shell.
I also will never forget the time I spent a day doing a perl script to parse some data... took some time, came back, and did it in a one line awk... Very powerful.
I think somewhere in here is room for discussion of data, text, and the unix philosophy as everything as a file.
Most sysadmins do not use python routinely. They work in the shell all day every day though.
I would argue that a well written Python script is going to be more comprehensible to the average sysadmin than an equivalent shell script, even assuming the sysadmin is more conversant in shell. More portable and testable (!!) too.
[0] http://www.catb.org/esr/writings/unix-koans/ten-thousand.htm...
Error handling is particularly problematic. If you know all the tricks (things like set -e, set -o pipefail, always capture stderr, etc.), it's not too bad, but there are still so many traps that the mental overhead of catching every single edge case is exhausting.
One nasty issue with Bash is quoting, escaping and word splitting, because almost everything in Bash is about string manipulation. There are so many edge cases: $FOO is not the same as "$FOO", and sometimes you have no choice but to turn it into an array with eval FOO=($FOO) if you need mimic Bash's tokenization behaviour (e.g. respect backslashes and quoting).
(Yes, Bash has arrays! A lot of scripts get simpler when you realize this.)
These days I reach for Ruby very quickly even for small things.
It's a bit like `checkbashisms` but a lot more expansive and with formal classification of the warnings.
Also, every rule in shellcheck has a corresponding github wiki page which describes the issues and common fixes. For example SC2002 (https://github.com/koalaman/shellcheck/wiki/SC2002)
EDIT: "and requires 2GB of RAM to compile." whaaat?
I'm very surprised you've been getting by with less than a gig. What's your arch and GHC version, if you don't mind?
Older GHC versions appeared to use significantly less RAM, and I imagine 32bit pointers would save a lot.
Your guess is right, I do my builds in Debian 8 with ghc and cabal from the repository. I do cabal update before compilation but I don't think this upgrades the ghc so the details should be like:
arch: i686
ghc: 7.6.3
I also use these arguments for cabal install but no idea if they affect anything:
--disable-documentation --enable-split-objs --ghc-options='-static -optc-static -optl-static -optl-pthread'I would like to propose a different moral of the story, which is, as soon as your shell script requires control flow beyond executing a series of commands one after another, switch to a better programming language, such as Python.
Clear syntax is better than pretty syntax.
I would no more want to do system administration with Python than I would want to write a web application server in bash.
And yes, `[[ ... ]]` is a bashism. It's not going work on the HP-UX box, that you do not have in your server fleet.
There are also a relatively large number of tcsh users in the world. This is why, even now on latest fedora, /etc/profile.d contains csh scripts.
As for sh, standards exist for a reason. Try to empathize with people who are not in your bubble.
That means dash & ash.
Portability can matter.
I wish. I've to take care not to break things like AIX, HP-UX, Solaris 10, Openbsd 5.1. That's an open source project, but there've been actual production customers complaining when we broke things in the past...
EDIT: corrected openbsd version, machine has been updated since the last complaint
[ -z $var ]
[ $var ]
I always just use [ "$var" ] and it's never failed me so far; should I be worried?
Context: I have the same feelings about Bash.
I also really like the pattern Django uses for extensible custom commands. I haven't built a system that extensive for my own apps yet but have enjoyed working with it when I am doing Django apps.
https://docs.djangoproject.com/en/1.11/howto/custom-manageme...
The other "problem" is quoting. In which case what would you expect to happen? Most people avoid spaces in file paths for this very issue. Argument passing is whitespace separated, so an application has no idea that you meant two (or more) args to be just one. Simple solution, use quotes.
If you were to shun all CLI usage in favor of another programming language, then you would be doing the exact same thing with system calls; Using the example from the article for consistency (even though it wouldn't make much sense in a more expressive language):
var = "some value";
exec("[", "-z", var, "]"); if var is not None:
...
else:
...
if not var:
...
if var == '':
...
That is a lot more readable to me than:
[ -z $var ]The Bash-ism `[[` (Compound Conditional) is an attempt to reduce the unintuitiveness of `[` or `test`. However, you would still need to quote "$var". The biggest pain point of using an external application like `[` is that `&&` is a shell construct. As you may know, `&&` means "if the previous program succeeds, then execute this next statement. But, if you do `[ -z "$var" && ...`, then `[` will always fail because the command lacks the terminating arg `]`. To do conditionals with `[` you would have to use `-a` for "and" or `-o` for "or". You would want to do either `[ -z "$foo" ] && [ -z "$bar" ]` or `[ -z "$foo" -a "$bar" ]`.
Conditional Expressions relieve those issues as `&&` and `||` will work between the braces. So, `[[ -z "$foo" && -z "$bar" ]]` is okay.
(Some other pain points to `test` are `=` tests for string equality, not assignment; although, `==` is a synonym. For numerical equality testing, there are `-eq`, `-ne`, `-lt`, `-gt`, etc. Then, for those unfamiliar with the Unix way (and even, to some extend, to those who are), there are three cryptic sets of tests for File Type, Access Permission, and File Characteristics.)
But, then, Conditional Expressions makes a few head scratching changes. Like `<` and `>` are for sorting strings; not numbers. And, instead of leaving `-a` and `-o` as-is, they were remapped so that `-a` is synonymous with `-e` (wat?) and that `-o` is for testing shell options. (Doing a little research tell me that `[[` comes from `ksh88`, so maybe they're there for parity.)
Manpage for `test`, `[`: https://www.mankier.com/1/test
Conditional Expressions: http://www.gnu.org/software/bash/manual/html_node/Bash-Condi...
At some point, I just started doing more and more in bash and started using Perl only specifically for anything I was having trouble getting working with sed+awk.
As of today it's probably been 3+ years since I wrote anything in Perl, I'll just drop to Python if I need it now. Also my sed and awk skills have gotten better.
[[ -z $var ]]
Edit to add: http://stackoverflow.com/questions/669452/is-preferable-over...
> %q ARGUMENT is printed in a format that can be reused as shell
> input, escaping non-printable characters with the proposed POSIX
> $'' syntax.
[ -z "$var" ]
you can even go further to do "${bar}", this is even safer I feel.
Most of the time, whatever is in [[ ]] can be easily emulated with POSIX/SUS-compatible tools and most of the time it doesn't matter that the tool is not a shell built-in.
Let's take people's favourite =~ operator. In bash, according to documentation, it uses ERE syntax. In zsh, it is ERE or PCRE, depending on an option being set. This is a subtle but important difference in how [[ ]] works. And then again, your environment can change the behaviour of a script you're running, which is (a) unexpected and (b) undesirable, and the change in the behaviour can range from "doesn't find thing it should" to "crashes" to "goes awry in an unexpected way".
[ -z "${var-}" ]
In Bourne once scripts I have been testing
test ${#var} -gt 0 and
test ${#var} -ge 1
test x"" = x$varAs an aside/self-plug, I wrote a blog post[0] complaining about my least-favorite Bash gotchas.
0. https://lambda.complex.rocks/2017/03/why-i-dont-like-bash/