Comment by sambe | Hacker News Reader

Comment by sambe | Hacker News Reader

Parent Full thread

The overhead of repeatedly launching subshells/processes to do simple operations can add up quickly, especially if it is happening in loops or in parallel. Yes, you shouldn't be using bash for performance, we all know that. But scripts often grow over time and suddenly are found to be slow/resource hogs. I have seen people demand that proper logging be added to a bash program and it then get minutes behind because of the overhead of all the processes called to do the logging. I was able to make that 10000x faster using pure bash.

As to why people like it: it often feels more natural when you are automating what you would type interactively. Unix pipes/coreutils/etc. also feel like a better fit when that automation is mostly about connecting other programs (it's the auxiliary stuff that you'd maybe want in pure bash). Reading the subprocess Python documentation does not exactly fill me with joy. I've heard libraries like Plumbum make it a bit neater - but then you have to ask why learn a bunch of new libraries when I already know bash? In the end it's about the best tool for the job. The danger with bash is going too far, especially if you don't actually know it very well.

My favourite way to log.. Redirect stdout and stderr ( &> ) into a named pipe ( >() ) running "tee" And get the redirect into the log file as well. `exec &> >(tee ${__DIR}/${DOC_LOCAL}/${LOG_LOCAL})`

Would you mind expanding on this with an example? I'm trying to improve performance of my bash logging

+1. My goal is to learn how to robustly do any logging whatsoever. Eager to learn new tricks.

Do you just want to log the scripts execution or do you want something more structured? If it's the former you can redirect the output from within the bash script with this (apologies for any condescension, I'm not familiar with your skill tree):

  if [ ! -t 1 ]; then
    exec > /my/log/file 2>&1
  fi

The if statements tests if your at an interactive prompt, if your not all output from the script get's redirected to /my/log/file. The above poster is instead redirecting into a subprocess " > (tee)" that will both print the output and log it.

It should be noted that often the bottleneck is the terminal itself, try running your scripts with a "> /dev/null" to suppress output and verify the slow part is actually the script.

So this line only redirects all script output to a file as well as ensure that that output makes it onto your screen at the same time.

It is unstructured only in the way you allow any running command within your script to dump their output.

I run most commands inside my scripts with `> /dev/null 2>&1` and then rely on exit codes to wrap structured information to be echoed out with this function:

echo_l(){echo;echo '--------';echo "${1}";echo '--------';echo}

Or functions such as this to populate a log:

log_cat(){echo "${1}" >> ${__LOG} }

But the tee named pipe is the winner.

PS. The __DOC_LOCAL and __DIR variables start with these magic variables below. These variables are a life saver and allow easy directory and file manipulation, they kind of setup a top-level context:

# Set magic variables for current file & dir

__DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

__FILE="${__DIR}/$(basename "${BASH_SOURCE[0]}")"

__SCRIPT="$(basename ${__FILE})"

__BASE="$(basename ${__FILE} .sh)"

__ROOT="$(cd "$(dirname "${__DIR}")" && pwd)"

Reply on news.ycombinator.com