Hidden gems of moreutils

Rule of thumb about shells:

  You'll never know for sure what shell is the default, so write your scripts to a minimum shell family, and encode the name of that shell family in the shebang.

The default shell, for either the entire system or an individual user, could be:

  - A Bourne shell
  - A C shell
  - A POSIX shell
  - A "modern" shell, like Bash, Zsh, Fish, Osh

Use the following shebangs to call the class of shell you expect. Start with /usr/bin/env to allow the system to locate the executable based on the current PATH environment variable rather than a fixed filesystem path.

  #!/usr/bin/env sh
  #              ^ should result in a Bourne-like shell, on modern systems, but could also be
  #                a POSIX shell (like Ash), which is not really backwards compatible

  #!/usr/bin/env csh
  #              ^ should result in a C shell

  #!/usr/bin/env bash
  #              ^ should result in a Bash shell, though the only version you should
  #                expect is version 3.2.57, the last GPLv2 version

  #!/usr/bin/env dash
  #              ^ you expect the system to have a specific shell. if your code depends
  #                on Dash features, do this. otherwise write your code using an earlier
  #                version of the original shell family, such as Bourne or POSIX.

If you need to pass a command-line argument to the shell command before interpreting your script, use the fixed filesystem path for the shell. Due to bugs in the way kernels execute scripts, all arguments after the initial shebang path may be sent as a single argument, which is probably not what you (or the shell command) expect. (e.g. use #!/bin/bash --posix instead of #!/usr/bin/env bash --posix as the latter may fail)

Different shells have different implementation details. For example, Bash tends to skew toward POSIX conformance by default, even if it conflicts with traditional Bourne shell behavior. Dash may try to implement POSIX semantics, but also has its own specific behavior incompatible with POSIX.

In order to get specific behavior (like POSIX emulation), you may need to add a flag like --posix, or set an environment variable like POSIXLY_CORRECT. For shells that support it, you can also detect the version of the shell running, and bail if the version isn't compatible with your script.

Here are some references of the differences between shells:

  - Comparison of command shells[1]
  - Major differences between Bash and Bourne shell[2]
  - Practical differences between Bash and Zsh[3]
  - Fundamental differences between mainstream \*NIX shells[4]

[1] https://en.wikipedia.org/wiki/Comparison_of_command_shells [2] https://www.gnu.org/software/bash/manual/html_node/Major-Dif... [3] https://apple.stackexchange.com/questions/361870/what-are-th... [4] https://unix.stackexchange.com/questions/3320/what-are-the-f...

Thanks a lot for the link, I needed an improved `ts` for a long time now.

Generic way: https://github.com/ebikt/progress.pl

Even if you don't pass this argument, you can poke most implementations of dd(1) with a certain POSIX signal, and they'll respond by printing a progress line.

On Linux, this is SIGUSR1, and you have to use kill(1) to send it.

On BSDs (incl. macOS), though, the signal dd listens for is instead called SIGINFO (which probably makes this make a lot more sense for why a process would have this response to it.) Shells/terminal emulators on these platforms emit SIGINFO when you type Ctrl+T into them!

(For a lot more useful info about this behavior: https://stuff-things.net/2016/04/06/that-one-stupid-dd-trick...)

Bonus fact not mentioned in the above article: dd used in the middle of a pipeline will still "hear" Ctrl+T and print progress, since signals generated by a shell (think: SIGINT from Ctrl+C) are propagated to all processes in the process group started by the command. Test it yourself:

    cat /dev/zero | dd count=10000000 bs=1024 | cat > /dev/null

For copying disks, an even more powerful tool is ddrescue. It works exactly like dd but writes its progress into a map-file that can be used to pause and resume the operation any time. It can also skip certain sectors known to be broken.

Super useful if you have a big slow disk that you have to copy under unreliable conditions. Saved my ass several times with old partially broken hard drives.

Yeah, I should've written "cannot rely on that alone and also check". I've worked with vi, later vim, for 34 years and read about it here first; ddg'ing for it doesn't give many hits.

Any reason not to just :q! ?

Thanks !

The parallelism isn't part of POSIX though (AFAIK), that's an extension by whoever wrote your xargs.

If what you really mean is that it's already installed on every machine you use, fair enough. But it's not strictly portable in some standards-based sense.

"rename" shares the same fate, there are several implantations, completely different from each other

Bash kinda broke seq for me since you can just write {99..0}

Less known maybe for most people who read HN, but you're right that a lot of shell scripting folks would know about it.