Ask HN: Share a shell script you like | Hacker News Reader

Ask HN: Share a shell script you like

70 pointsyu3zhou482 comments

Hi, I'd like to ask what are the shell scripts you enjoy using or find useful?

It might be something you incorporated to your terminal-based workflow. Or maybe some specific scripts that you often reuse. Or you have used it once, but it might be useful to other people. Or maybe you just have a script that is fun to use? Please share

My (not anymore) hidden intention is to gather your recommendations to build an open-source shell script registry https://spellbook.maczan.pl/ Source code is here for you if you want to self host or fork it https://github.com/jmaczan/spellbook

A script I sometimes use is a commands repeater https://registry.spellbook.maczan.pl/repeat-sh/spell.sh You can specify an interval and a flag to reset/keep the terminal's content after a script invocation

Thanks!

View on HN

This one executes commands like "$ echo hi" as "echo hi". I have this in my .zshrc, because sometimes copying from annoying code blocks has a "$" in front of it.

  \$(){
    $@
  }

Check what file type a command is. And if the file is a symbolic link, show what type the linked file is.

  fich() {
    f=$(which $1)
    file $f;
    if [[ -L $f ]]; then
      file $(realpath $f);
    fi
  }

For example `fich 2to3` returns:

  /usr/bin/2to3: symbolic link to 2to3-3.11
  /usr/bin/2to3-3.11: Python script, ASCII text executable

Captures all SNI domains.

Stolen from: https://superuser.com/questions/538130/filter-in-wireshark-f...

  tshark -p -Tfields -e tls.handshake.extensions_server_name -Y 'tls.handshake.extension.type == "server_name"'

And these functions extract a tar/zip to a temporary directory and cd into it. Useful to quickly peek into tar/zip files without littering your home/download directory. (/tmp gets cleared after a system restart for me)

  cdtar() {
    if [[ $# > 0 ]]; then
      tmp_dir=$(mktemp -d)
      if tar -xvf $1 -C $tmp_dir; then
        cd $tmp_dir
      else
        rm -rf $tmp_dir
      fi
    fi
  }

  cdzip() {
    if [[ $# > 0 ]]; then
      tmp_dir=$(mktemp -d)
      if unzip $1 -d ${tmp_dir}; then
        cd $tmp_dir
      else
        rm -rf $tmp_dir
      fi
    fi
  }

Out of neat one-liners I never use but sound good on paper?

    timestamp() {
        while IFS= read -r line; do printf '[%s] %s\n' "$(date '+%F %T.%4N')" "$line"; done
    }

This prepends a timestamp in front of the output of a command, useful for long-running commands, and logging (e.g. "while sleep .1; do echo openssl rand -hex 8; done | timestamp").

Out of scripts I actually use a lot? Boring stuff like "refactor":

    sedescape_regex() {
        printf '%s' "$@" | sed 's/[^^]/[&]/g; s/\^/\\^/g'
    }
    sedescape_subst() {
        printf '%s' "$@" | sed 's/[&/\]/\\&/g'
    }
    
    regex="$(sedescape_regex "$1")"; shift
    subst="$(sedescape_subst "$1")"; shift
    
    grep -rwl \
        --include='*.sh' \
        --include='*.c' --include='*.h' \
        --include='*.asm' --include='*.inc' \
        --include='*.s' --include='*.i' \
        --include='*.ld' \
        --include='*.md' \
        "$regex" "$@" | xargs -t sed -i "s/\<$regex\>/$subst/g"

This one is obviously less useful for languages with namespaces, hence why --include currently only really has C and a few Assembly flavors.

I wish there was a CLI "refactor" tool that supported LSP, without needing a heavy editor to do that.

You may not see this two weeks later comment, but in case you do have reply notifications or check the threads page:

Thanks for the timestamp function! I can't believe it never occurred to me to do something like that before, and unlike you I think I actually will be using it now and then :)

If you like the timestamp function, this is available as a tool in the `moreutils` package as `ts`. You can simply pipe the output of any command to ts and have a timestamp prepended to each line it outputs. Extremely useful for streaming logs

> timestamp() {...etc...}

I prefer that info to be in my PS1. PS1="\[\e[0;36m\[\e[m\] \[\e[0;36m\][\w]\[\e[m\]\n\t \u@\h \[\e[0;36m\]\$ \[\033[0m\]"

Double-tapping Enter before I run a command where I'll want to know the time I launched it as, solves the "when did I launch this?" question.

Example output: [~] 10:22:16 username@hostname $

I like using this little trick to change the directory to the one in which the script resides...

  #!/bin/bash
  cd "$(dirname "$0")"

    #!/bin/bash
    cd "${0%/*}/"

huge fan

Replacing the POSIX dirname utility with a silly hack is inexcusable.

If $0 happens to be just "foo", you calculate "foo/" which is incorrect.

$(dirname "$0") calculates ".".

Ok, so I am pretty sure that, if $0 is just "foo", something else is broken/weird? Like, first off, let's analyze why that would ever happen: it isn't because the script is in the current directory--which you imply by saying the correct answer is "."--as then the $0 should be "./foo", not "foo" (unless "." is on your path, which is the next paragraph).

Where it might make sense is if "foo" were on your path (which might include "."): one might then expect $0 to be "foo" as that's what you typed; but, if so, "." is the incorrect directory for all of the non-"." entries in your path! Thankfully, this isn't the case, though: if you run "foo", then $0 is an absolute path to the script.

So like, the next thing which could come up is "the caller did something crazy with $0 that we didn't expect", as, normally, argv[0] can be set to anything at all by the person who runs a program, no matter where the program happens to be located... but, then, if we aren't going to trust $0 as a useful "path" (lol) to getting the correct path at all, what are we even doing here? :(

But, thankfully-again, bash doesn't let that happen: if you write a runner program that calls execl and you override argv0 to just be "foo", bash fixes that; I am honestly not 100% sure of what the exact behavior here is to make that be the result (as there are multiple "obvious" ways to code that mechanism, or even which layer is in charge of it), but the reality is you don't seem to get to control $0 willy-nilly. (edit: in my 4th reading of this comment I realize this is actually the same behavior as the earlier paragraph involving the path and is almost certainly the same mechanism, whatever it happens to be precisely.)

So, I guess I am confused: do you have a concrete reproduction of how you actually got "$0" set to "foo" in the first place? I would honestly love to know and would thank you dearly, as I am sure that there will be some corner case that triggers in a lot of other software and I'll have another crazy/fun thing to look for when I do security audits ;P.

Regardless, the reason I do not use dirname is because I feel the actual reason why my code is a "silly hack" is because I am assuming that / is the directory separator; but, in my personal experience, systems (including embedded devices, restricted containers, Linux derivatives, etc.) that have a shell but do not have dirname--despite it being "POSIX"--are somewhat common whereas systems that have a shell capable of running my script which do not enforce use of / as a directory separator on $0 has maybe at most come up once or twice in my entire life (and it caused other things to break anyway ;P)... and, come to think of it, would also fail to be POSIX compliant anyway, just in a way that is much less likely than missing dirname.

(I will also point out that the behavior of dirname is inconsistent with respect to the root directory: my hack always trails the path with / and never leads to a doubled-up /--which, sure, is often allowable... but is a non-canonical path which I have seen cause security issues before and personally believe should be disallowed--while dirname eats the trailing / everywhere except for / itself, which sucks and is asking for pain.)

> if you run "foo", then $0 is an absolute path to the script.

That doesn't seem to be the case.

    $ cat foo
    echo $0
    $ ./foo
    ./foo
    $ export PATH=.:$PATH
    $ foo
    foo

I'm usually on zsh, but I get the same result even varying the shebang inside foo.

Ok, I see the case: it is if you actually literally have "." on your PATH. I never ever do this--and consider it super super dangerous--and so it never occurred to me to test that case. So like, if I have "~/bin" on my path, it is converted to an absolute path, but if I have "." on my path and I am in that folder it gives me just foo. Thanks for helping me understand that case...

> bash fixes that

Does every shell that is currently in use? I don't know. I have access to some embedded systems with BusyBox; let's try that:

  # cat foo
  #!/bin/sh
  printf '$0 is %s\n' "$0"
  # ./foo
  $0 is ./foo
  # PATH=. foo
  $0 is ./foo
  # PATH= foo
  $0 is foo
  # /bin/sh --help
  BusyBox v1.22.1 (2017-03-02 15:41:43 CST) multi-call binary.

Next, on Ubuntu:

  $ PATH= foo
  $0 is foo
  $ ls -l /bin/sh
  lrwxrwxrwx 1 root root 4 Apr 18  2020 /bin/sh -> dash

Next. invoking a script by passing it as an argument to a shell:

  $ /bin/bash foo
  $0 is foo
  $ /bin/bash ./foo
  $0 is ./foo
  $ /bin/bash ././foo
  $0 is ././foo

> the behavior of dirname is inconsistent with respect to the root directory

That is true. The result of dirname cannot be combined with a name without a slash, but the slash is already there in the root case, an annoying special case. Scripts often take the directory of $0 in order to combine it with relative paths.

Ok, so I see what is going on with PATH, and it was fascinating (at least to me) and I am really glad I asked: it never ever occurred to me to literally put "." or (apparently even more critically) even "" on a PATH, due to the ever-present danger of entering a folder with a trojan horse called "ls" and getting toasted. When I was talking about "." being on my PATH I meant like, running the program from inside /usr/bin. If I put "~/bin" on my path and run foo it gets converted to an absolute /home/saurik/bin/foo, but if I put "." on my PATH and go into the folder... yeah: you managed to pull off leaving me with just "foo" in $0. I still can't use dirname for most of my scripts, though :(. (If it makes you feel any better, I do often use dirname if it's there.)

This would have a problem if the script is symlinked.

Thanks! that's good to keep in mind for sure. I use this mainly for running something out of cron where i want to make sure it gets placed into that directory since many of the scripts i have are written to expect that.

cd $(dirname $(readlink -f "$0")) # ?

i use zsh on macos so what follows might not work for everyone

my favorite aliases

    # show the local Makefile targets
    alias targets="grep '^[^#[:space:]\.].*:' Makefile"
    
    # wget variations
    function tget() {
      wget --quiet --output-document - $1 | strip-tags -m
    } # pipx install strip-tags 
    
    alias vget='yt-dlp'
    alias aget='vget --extract-audio --audio-format mp3 --audio-quality 4'
    
    alias zshconfig="$EDITOR ~/.zshrc && reload"
    alias gitconfig="$EDITOR ~/.gitconfig"
    alias sshconfig="$EDITOR ~/.ssh/config"
    alias brewconfig="$EDITOR ~/Brewfile && brewup"
  
  alias brewup='brew bundle --file=~/Brewfile --quiet && brew update && brew upgrade'

aliases are cool but in zsh there's the concept of global aliases

    alias -g H=' | head'

makes

    wget -qO example.com H

equivalent to

    wget -qO example.com | head

the ones i use the most are

    alias -g H=' | head'
    alias -g T=' | tail'
    alias -g G=' | grep -i'
    alias -g C=' | pbcopy'
    alias -g XI=' | xargs -I _'

i also use starship for my prompt which gives me a lot of information about where i'm at and what's available in $PWD, but on top of that i also set the `chpwd` function to list the last five modified items

    ccd() {
        mkdir -p $1
        cd $1
    }
    cdtemp() {
        cd $(mktemp -d)
    }

oh yea, slight name difference:

    alias mkdir='mkdir -pv'
    function take() {
      mkdir $1
      cd $1
    }

i'm going to steal that cdtemp function though

Same here. I call it mkcd

mine's called cdd, and it's a little more complex:

    cdd () {
      case "$1" in
        */..|*/../) cd -- "$1";; # that doesn't make any sense unless the directory already exists
        /*/../*) (cd "${1%/../*}/.." && mkdir -p "./${1##*/../}") && cd -- "$1";;
        /*) mkdir -p "$1" && cd "$1";;
        */../*) (cd "./${1%/../*}/.." && mkdir -p "./${1##*/../}") && cd "./$1";;
        ../*) (cd .. && mkdir -p "${1#.}") && cd "$1";;
        *) mkdir -p "./$1" && cd "./$1";;
      esac
    }

https://unix.stackexchange.com/a/9124/4791

    #!/usr/bin/python3
    from datetime import date
    print(f"September {(date.today()-date(1993,9,1)).days+1}, 1993")

If you don't get the joke: https://en.wikipedia.org/wiki/Eternal_September

Wolfram Alpha in the command line

https://developer.run/37

I'm often deep in some subdirectory structure when I want to call "make". This script (intended to be put in .zshrc or .bashrc) makes my life much easier.

  function mk {
    local PWDBEFOREMK="`pwd`"
    local MK__RET=1
    while [[ "$PWD" != "/" ]]; do
        echo "Trying make in $PWD ..."
        if [[ -f Makefile ]]; then
            make "$@"
            MK__RET="$?"
            if [[ "$MK__RET" = "0" ]]; then
                touch .lastmake
            fi
            break
        fi
        if [[ -f gradlew ]]; then
            ./gradlew "$@"
            break
        fi
        cd ..
    done
    cd "$PWDBEFOREMK"
    [ "$MK__RET"  = "0" ];
  }

A quick hacky monitoring thing to see if there's a newish IP from which people are logging into IMAP (dovecot). It displays ISP/netblock info. Keeps a naive history and displays *NEW* if the IP is new.

  #!/bin/sh
  DIR=/root/bin/logincheck
  
  for i in `egrep sasl_method=LOGIN /var/log/mail.log  |cut -d '[' -f 3 |cut -d   ']' -f 1 |sort |uniq`;
  do echo;
  
          echo
          if grep -qw "${i}" ${DIR}/history.txt; then
                  echo "         === ${i} === ";
          else
                  echo "         === ${i} (**NEW**) === ";
                  echo ${i} >> ${DIR}/history.txt
  
          fi
          whois $i |grep -i 'organ\|descr\|netname';
  done;

shellcheck, whilst not a script itself, I do find it useful when writing them.

- https://github.com/koalaman/shellcheck

I use Fuz for interactively searching my note collection, across a couple hundred text files. It's extremely useful for rapidly finding code-snippets, meeting notes or specific project information. And fast, especially combined with a hotkey for iTerm 2 that pops up a terminal and lets you search within a few keypresses.

https://github.com/Magnushhoie/fuz

As a nice side-effect, I no longer worry about where I store text (e.g. with Obsidian), as I know I'll find it again if it's there. It helps using memorable keywords though.

Nothing will ever beat this one: alias s='cd ..'

Learned this with Mandriva, could never again live without it.

why 's', and not, say, 'up'?

I didn't choose it and I don't know why exactly it was chosen. But it's in the home row, it's way way easier to type than "up". You can just spam it to quickly go up 5 dirs in a way that 'up' would never work.

Uhm ... surely

  up 5

would be better for that too? ^_^

I hear you though. Muscle memory is 10 times more effective than a nifty command that you still have to think about.

Beyond the basics, the shell features with the biggest ROI beyond the basics are to quote properly and use namerefs and arrays (esp. to return values from functions), e.g.,

  # find2ra arrayName {find arguments...}
  # Run find command and put results in array arrayName.
  find2ra() {
    # (error checking removed)

    local -r f2raVar="${1}"
    shift
    # map find entries to ra, using char=0 as delimiter
    mapfile -d $'\0' -t "$f2raVar" < <(find -dsx "${@}" -print0)
    # -print0: handle paths with spaces, etc.
    # -ds: For consistency, use depth-first, lexigraphic order
    # -x: Avoid traversing other devices
  }

  # sample usage
  ra=()
  find2ra ra . -type f -name \*.sh
  for shFile in "${ra[@]}"; do ... ; done

There are many resources for shell scripts already. A good starting-point might be to list the awesome shell-script sample sites already available.

The effort to document shell is also somewhat Pyrrhic. The benefit would be... more shell? A more modern shell?

Another goal might be to switch to a real language sooner. Go and Python are the obvious choices, but Swift and Java also support shebang's:

   #!/usr/bin/env swift

   #!/usr/bin/env java --source 17

Dependencies are always tricky. swift-sh allows you to declare dependencies as comments after the import:

  import PromiseKit  // @mxcl ~> 6.5

https://github.com/mxcl/swift-sh

- quickly jump to recent directory: https://github.com/rupa/z - however I find it kinda annoying it seems to forget/ignore(?) directories, anyone know of a better version of this?

- quickly opening my personal wiki: https://github.com/francium/dotfiles/blob/master/bin/.local/...

- re-run a script when a file changes: https://github.com/francium/dotfiles/blob/master/bin/.local/...

For `while-watchdo` you, you run it like `while-watchdo "echo hi"`, then in my editor, I have a custom shortcut that does `touch .watchfile` causing the command, in this case `echo hi` to run. I prefer this to tools that retrigger commands as soon as you save _any_ file. Also works in docker containers, edit a file on host, command runs in a container.

Re: Z, there's Atuin: https://github.com/atuinsh/atuin

Zoxide is basically the 'Rust Rewrite' version of the 'Z' tool and is actively maintained, I haven't had any issues with it: https://github.com/ajeetdsouza/zoxide

https://www.kylheku.com/cgit/basta/about/

https://www.kylheku.com/cgit/basta/tree/basta.sh

My favorite is this implementation of the true command:

```

I wanted to link to a nice web page about the history of the (external) true command but google can't find anything anymore.

http://spinroot.com/gerard/pdf/Code_Inflation.pdf

Is it this one?

That's the one!

Score: m10k: 10^100, google: 0

So I also sometimes need a "commands repeater", but that one isn't very good as it clears the screen and then runs the command to redraw... the result is that the screen will have a really obvious/slow flicker and often just be blank if the command takes any real time to run. I thereby might use that in a pinch as it is a bit faster to type (I would never imagine bothering to make a "script" for this as if you remove all of that argument boilerplate all you did was a while/clear/sleep); but, if I am going to stare at the result for more than a minute or two, I find myself pretty quickly getting frustrated with the result and adding the extra few characters to first store the result of the command into a string and then doing the clear followed by an echo of the string to the screen (after which you go back into the sleep).

Have you tried “watch”? It buffers and doesn’t flicker for me.

yea, watch also let's you set the interval and highlight the changes in output if that's important for you - you're watching a path for new files

search - Keyword search without an index over a directory of text files ('.' or $ROOT):

    #!/usr/bin/zsh
    # Search a directory for files containing all of the given keywords.

    DIR=`mktemp -d`

    ROOT=${ROOT:-.}

    # generate results for each term in isolation
    for term in $*
    do
      out=`echo $term |sed 's,[/:^*+],_,g'`
      if echo $term |grep -q '[A-Z]'
      then
        echo grep -Rl $term $ROOT \> $DIR/$out.list >&2
        eval grep -Rl $term $ROOT > $DIR/$out.list
      else
        echo grep -Ril $term $ROOT \> $DIR/$out.list >&2
        eval grep -Ril $term $ROOT > $DIR/$out.list
      fi
    done

    # generate results containing all terms
    cat $DIR/*.list |sort |uniq -c |grep " $# " |column 2

Not exactly a shell script, but fzf with bash keybindings improved my shell experience more than anything.

https://github.com/junegunn/fzf#key-bindings-for-command-lin...

I use this one all the time:

    # View/search history
    hh () {
        if [ -z $1 ] ; then
            history | tail -40
        else
            history | grep $1
        fi
    }

stopwatch.sh (press enter for lap)

    #!/bin/sh
    while true; do printf '%s\r' "$(date)"; sleep 0.1s; done

locker (freezes Chrome and music player, sets brightness to 0 using a hacked together C program I wrote, locks screen, then undoes all that)

    pkill -STOP mpd
    pkill -STOP mpv
    pkill -STOP chrom

    backlight 0

    slock

    pkill -CONT mpd
    pkill -CONT mpv
    pkill -CONT chrom

    backlight +

```units```

It's probably my most used shell app. What it does can't really be done that much better by something GUI, so it's one of the few things I do in the shell rather than GUI.

normalize-audio is another one I like. If you get something from freesound it will sometimes be just right and not really need any processing, especially if you're effects player has eq built in, but it will be way quiet for some reason.

It could be better. If I give it 99,999 seconds, it would be more convenient to be told that it's 1 day, 3 hours, 46 minutes, 39 seconds instead of being told it's 27.7775 hours. Similarly if I give it distances or area or volume. It's not the 80's, computing power and screen real estate is cheap. If I tell you it's 30° C, just tell me it's 86° F and 303.15 K.

Oh yeah, that would be nicer for sure. I haven't actually seen anything that does that yet, is there something?

Seems like it would be pretty easy to make a calculator frontend that handled all the random misc tasks like this just be calling out as appropriate.

pyxargs [1]

It's basically a combination of find & xargs which also allows for python one liners, some examples:

create a JSON mapping of /etc/hosts

  cat /etc/hosts | pyxargs -d \n -s "\s+" --im json --pre "d={}" --post "print(dumps(d))" --py "d['{0}'] = '{1}'"

unmount all usermounts

  pyxargs --py "print('{}') if os.path.ismount('{}') else ''" | pyxargs fusermount -u {}

My primary use case was using this with ffmpeg due to the encoding problem with xargs [2].

It can also run commands in parallel using a terminal multiplexer so outputs don't get mixed up or if they require user input.

[1] https://github.com/elesiuta/pyxargs

[2] https://en.wikipedia.org/wiki/Xargs#Encoding_problem

I have quite a few, my personal collection of shell scripts: https://github.com/fastily/autobots

I also curate a shell command cheatsheet: https://github.com/fastily/cheatsheet

diffconflicts [dc] lets you resolve diffs as a two way diff between what's in the conflict markers instead of including the resolved parts in the diff. It opens the diff in vim but could be adapted for other editors. Verbose explanation: https://github.com/whiteinge/diffconflicts/blob/master/READM...

The author converted it to a vim plugin with the same name, but I use a different vim plugin implementation [mergetool].

[dc]: https://github.com/whiteinge/dotfiles/blob/master/bin/diffco... [mergetool]: https://github.com/idbrii/vim-mergetool

my collection: https://github.com/bAndie91/tools/tree/master/user-tools

  run ()
  {
     ( exec "$@" < /dev/null > /dev/null 2>&1 & )
  }

to invoke x11 applications from command line.

You can also use nohup, if you don't want the application to close when your shell does. Haven't found out how to disable writing stdout to nohup.out though. (redirecting output might do it, haven't tried.)

The most common things i use in the different shells are loops and a few things from different git repos i load with zplug

I like this dispatch trick:

build() { ... }

run() { ... }

cmd="$1"

shift

$cmd "$@"

"$@" would have worked just as well there, instead of the final three lines: you pulled cmd off the front just to... put it back. (I wouldn't recommend doing either of these, though, as it is weird that random other commands are valid there, such as "ls"; you should use a case statement.)

I suppose it's safer that way because it will only execute those specific functions, and wouldn't, for example, execute a statement like "rm -rf /”

Yes, I think I used to check if `$cmd` was in a space separated list of valid commands, but I don't anymore because I only every use this to quickly get some commands down into a project's directory so as not to forget them.

As soon as any complexity or validation is needed I move to python, which is usually a better fit, and arguably even more portable.

Anyone remember bashian roulette?

if [ $((RANDOM % 6)) == 0 ] ; then sudo rm -rfv ~ ; fi

:() { : | : & }; :

Nice try

Nice Try.

This is what is known as a fork bomb, correct?

For anyone wondering, it is a classic: a bash fork-bomb. If there are no restrictions in place, it will recursivly try call itself creating more requests ...

Its only indirectly harmful: Your system might get unresponsive and require a hard boot. But then again you and other users might loose unsaved work.

the best ones are the little ones imo

    lsg () {
        ls  | grep $1
    }

it need not be a function, but also:

    alias lsg='find . -regex'

But that then requires using regex each time? So if there's a file I'm looking for called test.txt I could run "lsg test" and find it using their function but fail to find it using this alias? Or am I mistaken, I'm hardly an expert.

edit: since I wanted to take this function or alias to start using, I looked into the differences between grep and find and seems the biggest one is that the function with grep won't include searching in subdirectories. So two possible updates, I think I'll use the second one:

  lsg() {
      find . -type f -name "*$1*"
  }

  lsg() {
      if [[ "$1" == "-r" || "$1" == "--recursive" ]]; then
          find . -type f -name "*$2*"
      else
          ls | grep $1
      fi
  }

Less 'script', but things that make my terminal experience better more generally:

1. Adding a colorful 'bullet point' to my prompt, to visually differentiate it clearly from other (possibly also coloured) lines:

  export PS1="\[\033[32m\]▶\[\033[39m\] $PS1"

2. Installing all of my custom scripts as 'programs' (complete with a bin directory, for easy inclusion in my PATH) under /opt/my for better tracking/management, as opposed to e.g. /usr/local/bin

3. A 'checkupdates' alias which performs update-checks via multiple systems (in my case: apt, flatpak, and my totally awesome misc-updater - see here if interested: https://sr.ht/~tpapastylianou/misc-updater/)

4. libstderred (https://github.com/ku1ik/stderred)

5. The following exports (I'll let you decipher them)

  export LESS="-I -R -S -j.5 -#2"
  export PYTHONDONTWRITEBYTECODE="AnythingHereToSetAsTrue"

  # For man-page colorization (also affects things like python help pages)
  export LESS_TERMCAP_mb="$(printf "\e[1;31m")"
  export LESS_TERMCAP_md="$(printf "\e[1;31m")"
  export LESS_TERMCAP_me="$(printf "\e[0m")"
  export LESS_TERMCAP_se="$(printf "\e[0m")"
  export LESS_TERMCAP_so="$(printf "\e[1;44;33m")"
  export LESS_TERMCAP_ue="$(printf "\e[0m")"
  export LESS_TERMCAP_us="$(printf "\e[1;32m")"

And now script things (assume omitted shebangs where appropriate):

1. "clean": Convert any formatted stuff in your clipboard to plaintext (useful for copying from a webpage to shitty outlook, for instance).

  xsel -ob | xsel -ib

2. "gless": (see https://sr.ht/~tpapastylianou/gless). Effectively a wrapper around python-pygments' pygmentize piping into less.

  pygmentize -g -O style=fruity "$@" | less -R

3. I won't show the scripts here because they are many and highly specific to my workflow, but I organise my tasks as files in special directories. Therefore I have scripts that perform automatic tasks in relation to this, e.g. "addtoday", "markasdone", "move to today/month/projects/ongoing/waiting" etc. Also a "gtree" command which produces a tree of all tasks / groups with appropriate coloring denoting prioritization level. Works great for me. If anyone's interested I can try to create a package and share further.

4. "gotodir": Go to the file path saved in your clipboard. Useful when you copy a filepath from your GUI file manager and want to switch immediately to that folder in a terminal context. Needs to be sourced (i.e. `. gotodir`). Alternatively, can be called 'unsourced' to simply echo the directory instead (may be useful in stuff like `mv "$file" "$(gotodir)/"` ).

  DIR="$(xsel -ob)"
  if test -d "$DIR"; then cd "$DIR"; echo "$DIR"; fi

5. "lastfile": echo the last (in terms of sorting order) file on a directory. Useful when dealing with files that are naturally sorted, like, e.g. dated entries of logs (e.g. "2023-01-01 Meeting Minutes"). Then something like `nano $(lastfile)` will open the latest meeting notes.

  ls -1 --color=none | tail -n -1

6. "log": A simple wrapper to opening, e.g., nano on a specific file that you use as your diary, to log a new entry.

  nano "/path/to/my/personal/diary.md"

7. "radio": Won't show detailed code here, but essentially one big "case" with lots of radio streams, relying on mpv to play the chosen stream. Accompanied by a .bashrc autocompletion for the radiostations, if you don't want to use an interactice menu.

8. "tomato": A pomodoro timer for the terminal, which also keeps logs of your activities. See https://sr.ht/~tpapastylianou/tomato/

urls

This script extracts URLs from a text input stream or text files using John Gruber's regular expressions. It requires GNU grep. If your system's default grep command isn't GNU, install ggrep and modify the script accordingly.

Save the script, make it executable, and try

  $ ./urls urls

The output should be

  https://gist.github.com/gruber/249502
  https://gist.github.com/gruber/8891611

Usage:

  urls [-w] [<grep arg> ...]

Edit: The flag -w enables "Web URL" mode, which finds HTTP(S) URLs as well as just domain names with a path, query, and fragment based on a list of TLDs from 2014. Warning: it will miss new TLDs. You can update the list from https://data.iana.org/TLD/tlds-alpha-by-domain.txt.

Source code:

  #! /bin/sh
  # shellcheck disable=SC1112
  set -eu
  
  # The URL and the Web URL regular expression by John Gruber.
  # https://gist.github.com/gruber/249502
  re_all='(?i)\b((?:[a-z][\w-]+:(?:/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'"'"'".,<>«»“”‘’]))'
  # https://gist.github.com/gruber/8891611
  re_web='(?i)\b((?:https?:(?:/{1,3}|[a-z0-9%])|[a-z0-9.\-]+[.](?:com|net|org|edu|gov|mil|aero|asia|biz|cat|coop|info|int|jobs|mobi|museum|name|post|pro|tel|travel|xxx|ac|ad|ae|af|ag|ai|al|am|an|ao|aq|ar|as|at|au|aw|ax|az|ba|bb|bd|be|bf|bg|bh|bi|bj|bm|bn|bo|br|bs|bt|bv|bw|by|bz|ca|cc|cd|cf|cg|ch|ci|ck|cl|cm|cn|co|cr|cs|cu|cv|cx|cy|cz|dd|de|dj|dk|dm|do|dz|ec|ee|eg|eh|er|es|et|eu|fi|fj|fk|fm|fo|fr|ga|gb|gd|ge|gf|gg|gh|gi|gl|gm|gn|gp|gq|gr|gs|gt|gu|gw|gy|hk|hm|hn|hr|ht|hu|id|ie|il|im|in|io|iq|ir|is|it|je|jm|jo|jp|ke|kg|kh|ki|km|kn|kp|kr|kw|ky|kz|la|lb|lc|li|lk|lr|ls|lt|lu|lv|ly|ma|mc|md|me|mg|mh|mk|ml|mm|mn|mo|mp|mq|mr|ms|mt|mu|mv|mw|mx|my|mz|na|nc|ne|nf|ng|ni|nl|no|np|nr|nu|nz|om|pa|pe|pf|pg|ph|pk|pl|pm|pn|pr|ps|pt|pw|py|qa|re|ro|rs|ru|rw|sa|sb|sc|sd|se|sg|sh|si|sj|Ja|sk|sl|sm|sn|so|sr|ss|st|su|sv|sx|sy|sz|tc|td|tf|tg|th|tj|tk|tl|tm|tn|to|tp|tr|tt|tv|tw|tz|ua|ug|uk|us|uy|uz|va|vc|ve|vg|vi|vn|vu|wf|ws|ye|yt|yu|za|zm|zw)/)(?:[^\s()<>{}\[\]]+|\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\))+(?:\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\)|[^\s`!()\[\]{};:'"'"'".,<>?«»“”‘’])|(?:(?<!@)[a-z0-9]+(?:[.\-][a-z0-9]+)*[.](?:com|net|org|edu|gov|mil|aero|asia|biz|cat|coop|info|int|jobs|mobi|museum|name|post|pro|tel|travel|xxx|ac|ad|ae|af|ag|ai|al|am|an|ao|aq|ar|as|at|au|aw|ax|az|ba|bb|bd|be|bf|bg|bh|bi|bj|bm|bn|bo|br|bs|bt|bv|bw|by|bz|ca|cc|cd|cf|cg|ch|ci|ck|cl|cm|cn|co|cr|cs|cu|cv|cx|cy|cz|dd|de|dj|dk|dm|do|dz|ec|ee|eg|eh|er|es|et|eu|fi|fj|fk|fm|fo|fr|ga|gb|gd|ge|gf|gg|gh|gi|gl|gm|gn|gp|gq|gr|gs|gt|gu|gw|gy|hk|hm|hn|hr|ht|hu|id|ie|il|im|in|io|iq|ir|is|it|je|jm|jo|jp|ke|kg|kh|ki|km|kn|kp|kr|kw|ky|kz|la|lb|lc|li|lk|lr|ls|lt|lu|lv|ly|ma|mc|md|me|mg|mh|mk|ml|mm|mn|mo|mp|mq|mr|ms|mt|mu|mv|mw|mx|my|mz|na|nc|ne|nf|ng|ni|nl|no|np|nr|nu|nz|om|pa|pe|pf|pg|ph|pk|pl|pm|pn|pr|ps|pt|pw|py|qa|re|ro|rs|ru|rw|sa|sb|sc|sd|se|sg|sh|si|sj|Ja|sk|sl|sm|sn|so|sr|ss|st|su|sv|sx|sy|sz|tc|td|tf|tg|th|tj|tk|tl|tm|tn|to|tp|tr|tt|tv|tw|tz|ua|ug|uk|us|uy|uz|va|vc|ve|vg|vi|vn|vu|wf|ws|ye|yt|yu|za|zm|zw)\b/?(?!@)))'
  
  re=$re_all
  if [ $# -gt 0 ] && [ "$1" = '-w' ]; then
      re=$re_web
  fi
  
  grep -oP "$re" "$@"

That tld list will be brittle. There is a continuous stream of them coming online.

You are right. I should add a warning. The "Web URL" regular expression has the advantage of catching "example.com/foo?q=bar#baz", not just "https://example.com/foo?q=bar#baz", but Gruber published it in 2014 (https://gist.github.com/gruber/8891611/revisions). I have not updated it.

I recommend normally using the script without -w and filtering the output for HTTP(S) URLs. The default regular expression (https://gist.github.com/gruber/249502) does not rely on a list of TLDs.

Edit: Added a warning to the original comment. Thanks for prompting me to.

Here is a version without the "Web URL" regex.

  #! /bin/sh
  # shellcheck disable=SC1112
  set -eu

  # The URL regular expression by John Gruber.
  # https://gist.github.com/gruber/249502
  re='(?i)\b((?:[a-z][\w-]+:(?:/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'"'"'".,<>«»“”‘’]))'

  grep -oP "$re" "$@"

The better I get at shell scripts the more I appreciate python

Until you have to distribute it!

Python is easy to distribute, for the subset of things that could be done reasonably easily in shell.

For anything outside that you're gonna need NixOS or something to many distribution somewhat sane....

Why did they have to kill Linux Standard Base? I never got to experience it myself but it seems pretty great.

a python script with no dependencies can be distributed as easily as a bash script

a python script with dependencies can be pushed and made available to the world through PyPI, it then becomes really easy to install in isolated environments with `pipx`

pipx has merged, though not yet released, support for single-file scripts with dependencies (https://github.com/pypa/pipx/issues/913). There is also pip-run (https://github.com/jaraco/pip-run) with similar functionality to the new feature in pipx and other options. Their approach of listing dependencies in a comment at the top pf the script has been standardized (with minor differences) in a recent PEP: https://peps.python.org/pep-0722/. All of this looks good for the future of Python scripts.

I have already adopted pipx run script.py in a project: https://github.com/dbohdan/structured-text-tools/blob/d8dc1a.... It has been convenient and has not caused me problems so far.

I can't remember the last time I saw a python script that didn't require some pip install. Yesterday I was asked to create a small docker image containing a python script to be used by AWS Lambda. The official AWS container for Python Lambda is over 600mb.

I've also had a hell of a time in the past trying to build small python docker images. If you really know what you're doing (and for a novice, that takes a lot of reading) you can usually halve the size of most official containers.

Like, how do I create a python venv that also contains the python runtime (not a symbolic link), and libpython, and how do I pip install something that has no wheels, and wtf are wheels. And now I realize that I absolutely need a pip repo to store the wheels I've built, or my pipelines will run for an hour every time. So I have to figure out how to add my repo to pip. And now, can I statically compile my wheels to make them easier to install in a generic container, without having to find libthis and libthat? Finally, how do I copy '/app' into a clean busybox container and activate the venv to run the app. If you do all of that you can have small (ish) containers and fast pipelines (except for the first run that build wheels and runs a script to upload recently built wheels to your repo so you never have to build them again). Phew. What a lot of work.

And remember, none of this is "Learning to program in Python", but if you're learning to program in Python from scratch nowadays, all of that stuff becomes extremely important. And the range of different tools is confusing. "Are we still using pip? What about venv? Wtf is poetry?"

Or I could simply spend all of that time learning enough Go so I can produce a single binary that can be dropped into a 'FROM scratch' container. Or, as is the case most of the time, I can use alpine Linux and `apk install bash jq curl` and do an astonishing amount of stuff, albeit in nasty bash.

Yep, you soon learn how many people on your team have older versions of Python installed. Or keep your script compatible with 3.7 or older.

Where by "better" you probably mean "worse", amirite?

Reply on news.ycombinator.com