Hacker News Reader

Comment by hedora | Hacker News Reader

I’ll bite. Show me the equivalent of this in python, including error handling, parallelism, disk footprint, and real-time status output:

   #!/bin/bash -eux -o pipefail

   TMPDIR=$(mktemp -d tempdirXXXXX)
   trap “rm -rf $TMPDIR” SIGINT SIGTERM ERR EXIT
   (
      cd $TMPDIR
      foo | grep x > foo.out & 
      bar > bar.out & 
      baz $TMP | tee baz.out &
   )
   wait

   tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh server.com bash -c “cat - > out.tag.zstd”

View on HN

    $ shellcheck myscript
     
    Line 1:
    #!/bin/bash -eux -o pipefail
    ^-- SC2096 (error): On most OS, shebangs can only specify a single parameter.
     
    Line 4:
    trap "rm -rf $TMPDIR" SIGINT SIGTERM ERR EXIT
                 ^-- SC2064 (warning): Use single quotes, otherwise this expands now rather than when signalled.
     
    Line 6:
          cd $TMPDIR
             ^-- SC2086 (info): Double quote to prevent globbing and word splitting.

    Did you mean: (apply this, apply all SC2086)
          cd "$TMPDIR"
     
    Line 9:
          baz $TMP | tee baz.out &
              ^-- SC2086 (info): Double quote to prevent globbing and word splitting.

    Did you mean: (apply this, apply all SC2086)
          baz "$TMP" | tee baz.out &
     
    Line 13:
    tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh server.com bash -c "cat - > out.tag.zstd"
                 ^-- SC2086 (info): Double quote to prevent globbing and word splitting.
                 ^-- SC2153 (info): Possible misspelling: TEMPDIR may not be assigned. Did you mean TMPDIR?

    Did you mean: (apply this, apply all SC2086)
    tar -cf - -C "$TEMPDIR" | zstd -6 -z - | pv | ssh server.com bash -c "cat - > out.tag.zstd"

You can forego the final "bash -c" by leveraging dd(1) to avoid the redirection problem:

    ... | ssh server.com dd of=out.tag.zstd

dd(1) reads from STDIN by default unless given an "if=" parameter, and writes to STDOUT by default umless given an "of=" parameter.

Thanks! Will definitely use that moving forward.

Python wins the triathlon by being second best in every game. Piping things between programs is about the only thing bash is good at, nothing else, of course it is going to look good when unconditional straight piping things through standard programs is the only requirement. It's like saying c++ is fastest at calculating pi, duh, nobody will argue against that. Now, instead, put that script into a bigger context where some actual program logic or other integrations are needed. It is going to explode in unreadability, bugs and only supporting happy-cases. It is only simple if your original problem is simple.

I like both Bash and Python, so I think I'll give this a try. Should I implement the bugs as is, or fix those first?

Reply on news.ycombinator.com