Pipefail: A missing shell option slowed Cloudflare down | Hacker News Reader

Pipefail: A missing shell option slowed Cloudflare down

blog.cloudflare.com

The post covers a lot of interesting Cloudflare tech and the story of the incident.

As for the punchline:

  dos-make-addr-conf | dosctr set template_vars -

> This cron job is implemented using a small Bash script. If you’re unfamiliar with Bash (particularly shell pipelining), here’s what it does:

> First, the dos-make-addr-conf executable runs. Its job is to query the Addressing API for various bits of JSON data and serialize it into a Toml document. Afterward, that Toml is “piped” as input into the dosctl executable, whose job is to simply write it into a Quicksilver key called template_vars.

> Can you spot the bug? Here’s a hint: what happens if dos-make-addr-conf fails for some reason and exits with a non-zero error code? It turns out that, by default, the shell pipeline ignores the error code and continues executing the next command! This means that the output of dos-make-addr-conf (which could be empty) gets unconditionally piped into dosctl and used as the value of the template_vars key, regardless of whether dos-make-addr-conf succeeded or failed.

The 'fix' from the post is to add to this script:

  set -e -u -o pipefail

I would suggest that choosing a pipe communication format where no content is valid input isn't very wise. There's a reason why we have machine-readable vs human-readable favoured formats.

At least have the dosctr program recognise empty input as a special case and reject it, if it's never what you want. This post seems to be a long 'first cause' analysis rather than root cause(s).

I'm confused about how this change actually fixes the problem.

The article says:

> Enabling this option changes the shell’s behavior so that, when any command in a pipeline series fails, the entire pipeline stops processing.

This is not true! The entire pipeline runs to completion just as it normally would. The only thing that is affected is the exit status of the pipeline as seen by the shell.

You can see this by running something like `(set -o pipefail; false | sleep 10)` and see that it still takes 10 seconds to complete even those the `false` command exits with an error immediately.

Even after turning on the pipefail option, running a command like `false | dosctl set template_vars -` will still set template_vars to the empty string.

Author here – you are correct, I tried to golf a relatively long script down to one line, and I failed :) The post has been updated.

If the update is:

set -e -u -o pipefail

(dos-make-addr-conf | tee config.toml) && dosctr set template_vars config.toml

well....

`tee config.toml` will still produce a empty config.toml

(set -e - u -o pipefail;false | tee config.toml) && cat config.toml

But you can use the '>' operator, that will create the file only if the command runs successful:

dos-make-addr-conf > config.toml && dosctr set template_vars config.toml

> But you can use the '>' operator, that will create the file only if the command runs successful:

It will still create the file.

Run this in bash, zsh or fish:

    false > falsefile

It will create an empty file called "falsefile". This is because the shell opens the file before the program runs.

What fixes it in your script is the `&&`. That causes the `dosctr` to only be run if the `dos-make-addr-conf` succeeded.

You are right. My bad! Thanks.

One other thing which could have prevented the shell failure is checking shell scripts with https://www.shellcheck.net/. It helps catch common errors and places where shell errors can fail either silently (like here), or catastrophically (`rm -rf $abc/` where $abc is undefined)

Perhaps that was also part of Cloudflare’s mitigations here too but not mentioned.

Very cool how Cloudflare actively looks for post-mortems they can report on, whereas other big companies you'll be lucky to get one

Reply on news.ycombinator.com