Hacker News Reader

Comment by pdimitar | Hacker News Reader

That's why I plan on migrating all my shell scripts to Golang programs f.ex. using https://github.com/bitfield/script -- it already has a number of simulations of shell commands and I'd contribute others if I had the time.

sh / bash / zsh scripts are just fragile and that's the inconvenient truth. People who devised the shell interpreters had good intentions but ultimately their creations grew to a scope 1000x bigger than they intended, hence all the "do X but if Y flag is set then do Z... unless flag A is also set in which case do Y and part of Z".

It's horrendous and I seriously don't get what's so difficult in just coding these scripts in a programming language that provides single statically linked binaries (like Golang) and just distribute it with your images -- or run them in CI/CD and have init containers and never include them in the images in the first place.

Inertia, of course. But I'll be actively working against it until I retire.

View on HN

> It's horrendous and I seriously don't get what's so difficult in just coding these scripts in a programming language that provides single statically linked binaries (like Golang) and just distribute it with your images -- or run them in CI/CD and have init containers and never include them in the images in the first place.

It takes longer to develop (for sufficiently small scripts), it's harder to verify, it's harder to debug, it requires a build step and it's a lot more effort to modify it in-place. The process for calling other programs is also extremely streamlined, making it perfect for integration tasks like CI/CD pipelines.

Some of these issues don't exist when using an interpreted language like Python instead, but this comes with its own problems. Shell is pretty universal, well-known and is, for sufficiently small problems, the quickest solution.

Python's facilities for calling subprocesses are pretty inconvenient compared to bash IMO. It defaults to binary output instead of UTF-8 so I almost always have to set an option for that. I wind up having to define threads in order to run programs in the background and do anything with their output in real time, which has an awkward syntax. The APIs for checking the exit code vs raising an error are pretty non-obvious and I have to look them up every time. And I always wind up having to write some boilerplate code to strip whitespace from the end of each line and filter out empty lines, like p.stdout.rstrip().split('\n') which can be subtly incorrect depending on what program I'm invoking.

It looks like you used some old tutorials?

"subprocess.run" appeared in python 3.5, and it's pretty nice - for example you so "check=True" to raise on error exit code, and omit it if you want to check exit code yourself. And to get text output you put "text=True" (or encoding="utf-8" if you are unsure what the system encoding is)

As for your boilerplate, it seems "p.stdout.splitlines()" is what you want? it's what you normally want to use to parse process output line-by-line

The background process is the hardest part, but for the most common case, you don't need any thread:

     proc = subprocess.Popen(["slow-app", "arg"], stdout=subprocess.PIPE, text=True)
     for line in proc.stdout:
          print("slow-app said:", line.rstrip())
     print("slow-app finished, exit code", proc.wait())

sadly if you need to parse multiple streams, threads are often the easiest.

My team is also in the process of refactoring a number of shell script CI pipelines into Go. With `go run` it might as well be interpreted, from a UX standpoint it’s as easy to call as a shell script but with the benefit of static types and a robust ecosystem of supporting libraries and not having to sed/awk your way through command output.

> It takes longer to develop (for sufficiently small scripts)

Obviously, but at one point I sat down and tried to estimate how much time I wasted on using `set -euo pipefail` in my scripts and still having to chase after silent failures. I might be biased at this point but it still seemed quite a lot.

For a lot of one-off tasks shell scripting is 90% superior (unless you are super comfy with Golang) because it takes literal minutes to write and then iterate on it. Sure. But my threshold for when to reach for a proper program has been getting lower and lower lately because I very quickly arrive at a point where I need typing, better error-handling, retries and a few others.

Not everyone's case, surely. It seems my journey was more along the lines of "I was using shell scripting much more than I had to and I am making a partial comeback to proper programs".

Eh, looking at your go library vs shell script, I'd choose shell script anytime. Here are major reasons:

- Debugging. I can prefix any shell script with "bash -x" and I'll get an step-by-step output of every command executed, including the arguments. I can copy-paste any part of script into interactive shell so I can run it over and over until I get things right.

- Error reporting: as long as you use shell's best practices, shell script automatically stops at a first error, and in most case there will be a clear error message printed. In golang, this is all manual, and it is pretty easy to ignore errors. (python really helps here)

(note your library seems even worse than that: not only it starts with go's error reporting handicap, you also designed it so "we just won't see any output from this program if the server returns an error response.", _and_ you actively mix stdout and stderr... this basically means any programs using your library will be nightmare to debug)

- Performance: "sort" is an external process, but it is designed to be able to sort even the data which does not fit into memory (it has spill-to-disk code). "grep" actually uses extreme optimization and things like Boyer-Moore algorithm - does your code do that?

(that said, shell scripts rapidly become unreadable after some complexity threshold.. my personal rule is "if you ever feel tempted to create common library of shell functions, it means your script become too complicated and it's time to rewrite it in real language ASAP". But until you hit that complexity limit, the shell scripts are very nice)

Valid criticisms but I should note I didn't write the library. Apologies if I left another impression.

I agree shell scripts are still more convenient for iteration but as you pointed out in your last paragraph, most of mine crossed the complexity threshold and it's now more worth it to me to invest in a proper program than to keep stumbling on poorly documented edge cases in bash/zsh.

And I again agree with your criticisms of the Golang library (worthy of filling a few issues in their GitHub repo btw, if you have the time), it's just that my stance there is: sure it will hurt replacing shell scripts but we should start getting to it. Being stuck in this local maxima is limiting.

Reply on news.ycombinator.com