Hacker News Reader

Comment by nneonneo | Hacker News Reader

This article promotes bad practices for dealing with SIGPIPE.

1. Closing stderr in Python is not a good idea because that’ll swallow any other errors that occur at exit. Redirecting stdout to devnull is really just a way to prevent the flushed output from going to the now-closed stdout and triggering another SIGPIPE. That’s more preferable than closing stderr and losing error output at exit.

2. Ignoring SIGPIPE is a terrible idea for a process that should do stream processing. Try making a yes clone and ignoring SIGPIPE - your process will likely run forever trying to shove “y” into a closed pipe. There’s a reason SIGPIPE was invented! Very few programs bother to check the return value from write/printf/etc.

View on HN

Agreed. I like articles like this because they show the learning process, which I think is super valuable. However, they really need a big disclaimer stating the author is experimenting and doesn't know the correct answer. Otherwise people stumble upon it and take it as authoritative.

Great point, I should probably add that disclaimer!

Agreed. Correct handling of SIGPIPE is quite subtle and often done incorrectly. I've some notes on SIGPIPE considerations at:

http://www.pixelbeat.org/programming/sigpipe_handling.html

> 1. Closing stderr in Python is not a good idea because that’ll swallow any other errors that occur at exit. Redirecting stdout to devnull is really just a way to prevent the flushed output from going to the now-closed stdout and triggering another SIGPIPE. That’s more preferable than closing stderr and losing error output at exit.

I don't follow. The standard for EPIPE/SIGPIPE handling is to silently exit with an error status. It's fine to close stderr to prevent spurious warning messages about flushing stdout.

> 2. Ignoring SIGPIPE is a terrible idea for a process that should do stream processing. Try making a yes clone and ignoring SIGPIPE - your process will likely run forever trying to shove “y” into a closed pipe. There’s a reason SIGPIPE was invented! Very few programs bother to check the return value from write/printf/etc.

Programs can correctly handle lost pipes masking SIGPIPE entirely, with error checking alone. Python's BrokenPipeError is raised on the basis of EPIPE, not SIGPIPE.

Re: programs not checking error returns of write() and close(): that is not really true in a language like Python with exceptions raised on IO errors. It always does the check, and the unwinder aborts the program if nothing handles the error. Sigpipe is completely unnecessary for Python programs. (It's also not necessary for C programs, but I guess AT&T didn't want to fix their programs to check for errors.)

The suppression of SIGPIPE was done in the Go code, not in the Python code.

Does the POSIX standard mandate that programs receiving EPIPE/SIGPIPE die silently? I don’t know of such a rule, and there’s plenty of programs that violate this. Python is a bit too verbose with the errors (with a full trace back and two copies of the error) so suppressing those errors somehow seems like a good idea for a general-purpose command line tool.

Does the POSIX standard mandate that programs receiving EPIPE/SIGPIPE die silently?

https://pubs.opengroup.org/onlinepubs/7908799/xsh/signal.h.h...

Yes. The default action of SIGPIPE is to terminate the process.

The standard just specifies that (a) SIGPIPE exists, (b) when it should be raised by the OS, and (c) what the default action is.

It does not specify that programs that mask or block or handle SIGPIPE/EPIPE must be silent when they do so.

> The suppression of SIGPIPE was done in the Go code, not in the Python code.

What? Your earlier comment I was responding to mentioned only Python, not Go. And Python suppresses SIGPIPE:

https://github.com/python/cpython/blob/master/Python/pylifec... ,

https://github.com/python/cpython/blob/master/Doc/library/si...

> Does the POSIX standard mandate that programs receiving EPIPE/SIGPIPE die silently?

Not to my knowledge.

Python "knows what it's doing" in the sense that it guarantees that it checks return values from write() and has an exception mechanism to ensure that errors aren't accidentally ignored. Therefore, they can get away with ignoring SIGPIPE because they have the appropriate error checking in place.

If you check the linked article, you'll see that it discusses both Python and Go, and mentions suppressing SIGPIPE in the Go code. My original comment did not mention Python for point #2; it was more of a general admonition. (And, like all Internet advice - there are always exceptions if you know what you're doing - it was more of a way to head off newbies who might see signal(SIGPIPE, SIG_IGN) and think it's a good idea!)

Useful to know is the exception is printed to stderr, not stdout. So if you continue the pipe after head you still get expected results.

cat doesn't die 100% silently either, check $PIPESTATUS after piping cat into shorter head and you will see its exit code is non zero.

The biggest reason I can think of not to close stderr (or any other of the standard handles) is that the next open(2) call is likely to get that same descriptor recycled. So now joe random open file is going to receive all the error messages from random libraries, or perhaps even from unrelated programs you may have forked with that file as fd 2.

The biggest takeaway here IMO is that Python breaks the standard contract regarding signals–at least for SIGPIPE. Python should not be catching and throwing an exception for SIGPIPE; it should simply exit immediately, which is literally the default POSIX behaviour... unless a script/program/process specifically installs a signal handler to perform cleanup before exit. Python has some pretty awful behaviours built into it, and this is one of them.

Half of this article is not "How do Unix pipes work", but "how to fix broken SIGPIPE handling in Python".

Python doesn't catch SIGPIPE, it ignores it.

The exception is raised from a -1/EPIPE return from libc write().

I fully agree that Python is often a bad citizen in terms of signal handling — it wants to only process signals on 'the main thread', but also wants end-users to fully control signal-handling. The two ideas are sort of at-odds and in general I find handling signals in Python frustrating.

Another bad practice, IMHO, is not to quit the program once the exception is thrown. Instead, the loop continues and the rest of the input gets fed to /dev/null

Thanks! So the preferred way is to redirect to dev null?

It's better to just do nothing and allow SIGPIPE to kill your program. That is the reason this signal exists. Python is not a good unix citizen in this case. Compare it to Perl for example where nothing special is required to do the right thing:

    $ perl -E 'say "y" while 1' | head -1
    y
    $

I see

> Python is not a good unix citizen in this case. Compare it to Perl for example where nothing special is required to do the right thing

How is this better? At least with the exception you can cleanly terminate your program if you need to.

In the unix pipeline model, components read input, perform some transformation, and print output. There is no need to do any clean-up. As nneonneo explained, because they rely on SIGPIPE, most unix programs don't even need to check for errors from write(2).

Naturally, in the rare case you want to do some clean-up you can:

    $ perl -E '$SIG{PIPE} = sub { die "cleanup" }; say "y" while 1' | head -1
    y
    cleanup at -e line 1.

I disagree. Saying python is "not a good Unix citizen" here is akin to saying cars are a bad highway citizen because people can crash them.

Python provides mechanisms to handle signals. The point of a signal is to indicate something outside of your process has happened and you may want to respond to it. It's up to the program to respond to relevant signals and Python in no way stops a developer from doing that.

> Saying python is "not a good Unix citizen" here is akin to saying cars are a bad highway citizen because people can crash them.

I don't follow your analogy. The default signal disposition for SIGPIPE is to silently terminate the process. Normally processes compose nicely on the command-line and this requires no extra work from the programmer. By disregarding this convention, most Python programs pollute your terminal when used in pipelines, which is why I claim that in this case Python is a bad unix citizen.

Yep, I follow your logic. As you say, in other languages it "requires no extra work", it's more a matter of convenience.

Python in no way stops you from a having a typical response to SIGPIPE, it's as simple as handling the exception and doing a sys.exit(), but it doesn't "just happen". This makes the typical path a little more contrived, but I don't think having explicit handling makes it a bad citizen.

To torture my analogy, in my mind a "bad citizen" of the highway may be a car incapable of doing the minimum speed limit, whereas Python just has a manual transmission vs Perl's automatic (in this case).

I'm splitting hairs though, your point is fair. I just think your objection is a little strong for what comes down to convenience.

It's not a matter of "convenience" any more, when you know for a fact that nobody dashing off quick five-line scripts in your language is going to bother doing the extra thing, and so the default behavior will be the only behavior for those scripts.

Defaults matter. It's like the difference between opt-in and opt-out for organ donation, or the difference between encouraging and requiring cars to have airbags.

Mosst everyone agrees that defaults matter, but not what the defaults should be.

Not an expert is no way, so I could be wrong, but in python could you use signal.SIG_DFL to have that expected behavior:

  # cat.py
  import sys
  from signal import signal, SIGPIPE, SIG_DFL
  signal(SIGPIPE,SIG_DFL)
  
  for line in sys.stdin:
      print(line)

True, but most Python programs don't do this, so they pollute your terminal with error messages when you attempt to use them as interior processes in a unix pipeline.

That seems to be discouraged by the official python docs.[0]

[0] https://docs.python.org/3/library/signal.html#note-on-sigpip...

> sys.exit(1) # Python exits with error code 1 on EPIPE

Should be 141 instead of 1. Convention is that when a program dies because of a signal, the exit code should be 128 + the signal number, in this case 13. So, 128 + 13 = 141.

Here are some examples:

  $ ( yes; >&2 echo $? ) | head -0 
  141
  $ ( ping localhost; >&2 echo $? ) | head -0 
  141
  $ ( perl -E 'say "y" while 1'; >&2 echo $? ) | head -0 
  141

In python, when using signal(SIGPIPE, SIG_DFL), you get the correct behaviour (at least regarding the exit status):

  $ ( python -c '                                                  
  import sys                                                                    
  from signal import signal, SIGPIPE, SIG_DFL     
  signal(SIGPIPE,SIG_DFL)                    
  while True: print("y");                         
  ' ; >&2 echo $? ) | head -0
  141

> Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid BrokenPipeError. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it.

Regarding that socket behavior, if that's the standard in other programming environments, why is it an issue from python's perspective? Doesn't seem worth eschewing standard unix conventions. I mean, it says "unexpectedly", but isn't it actually the expected behavior? Unexpected is for python to say that what's default (SIG_DFL) is unexpected.

> Convention is that when a program dies because of a signal, the exit code should be 128 + the signal number

Note that this is incorrect. This is a common misconception about POSIX/UNIX. Not every process exits with an exit code; that's right, a process can exit without an integer exit code! A process exits with either a) an exit code, _OR_ b) the signal which terminated the program (as a separate status in the underlying struct, rather than as an exit code).

Many shells–including bash–provide an abstraction that translates process exits caused by signals into 128+signo, but this is only to fake an exit code for processes as seen when executed within the shell; this is a nonstandard and shell-specific abstraction.

Look at the man page for waitpid() for details. There are macros to test for the difference between an exit status with a code vs. an exit due to a signal; eg. WIFEXITED() and WIFSIGNALED(). WEXITSTATUS() will return the exit status... if one exists because WIFEXITED() tests truthful. Hint: if you're used to seeing your shell report 128+signo, that's because your particular shell happens to check waitpid() for WIFSIGNALED() and sets $? to 128+signo for you… as an abstracted convenience.

tldr; An exit code, vs. an exit caused by a signal, are mutually exclusive statuses. Processes terminated directly by a signal do not have an integer exit code; your shell may provide a fabricated exit code to make things appear simpler, but that is shell-specific and NOT a standard convention provided by POSIX or UNIX.

You're right. waitpid(2) does document this.

I was also surprised to find that, at least on my linux system, the status integer returned by wait encodes the signal number in the least significant bits and the exit status in the most significant bits. So, when a process dies from SIGPIPE, the status (as set by wait(&status)) is 13; when it does exit(1), it's 256; when it does exit(2), it's 512; when it does exit(3), it's 768, etc. Maybe that encoding was done to avoid misuse from people seeing their exit codes in the raw status returned by wait, and thinking they didn't need to use the macros.

In any case, this does mean that, indeed, while the shell does display the same status for both `yes | head -0` and `(exit 141)`, they are in fact different.

TIL

That's why we love Unix shell: it takes everything we type and removes the types. :-)

In the exception handler, you should reset SIGPIPE disposition and reraise the signal:

   signal.signal(signal.SIGPIPE, signal.SIG_DFL)
   os.kill(os.getpid(), signal.SIGPIPE)

(Then there's no need to fiddle with stderr.)

It also gives rationale. The rationale is ... unconvincing.

This is bad advice, if you are doing anything other than output to stdout/stderr (which you almost assuredly are, unless you are doing something very simple like "yes"), you want to switch to /dev/null. For instance, running

  rm -vfr folder/ | head

involves SIGPIPE causing problems because it may or may not kill rm before it finishes deleting the directory, based on its internal output buffer size.

The problem here is connecting a program you want to run to completion to the head command, not how rm handles SIGPIPE.

head is just an example, if you pipe rm -vfr (or any program that does more than just output) into _anything_ the reasonable and default behavior should be to run rm to completion. (And from a programmer's POV, the reasonably thing is for syscalls to handle errors by returning an error, the way every other error is handled.)

Wouldn't it be better to be explicit? If intention is to "display first 10 lines, at most; discard the rest", you can write:

  rm -vrf directory/ | (head; cat >/dev/null)

I guess it depends on your semantics/mental model of what a pipeline should do. My mental model says that each part should run fully, and the pipeline should emulate sequential behavior, with "yes" being a bit of a hack in this regard. Cases where the behavior of a pipeline depends on the internal buffer size and operation ordering (e.g. what if rm decided to print everything it did only after removing everything?) seem like bugs to me. Maybe your mental model is different?

In any case, I think that in the vast majority of pipelines, the default SIGPIPE behavior is not what is desired.

Piping output of an "rm -v" is an oddball, with all the possible stdio buffering, I agree.

More real-life example fitting your model would be "start this long-running process and show me first 10 lines of output, just to make sure it didn't bail right away", but then, a more common pattern would be to keep a copy of the whole log too:

  myfoo >/var/log/foo & tail -f /var/log/foo | head

Counter-example would be "yes", but also unordered sampling. Let's say I want to know if there is file with "foo" in its name somewhere in a very large directory, just a yes/no. Continuing directory traversal after finding one is a waste of time. SIGPIPE is desirable here as well:

  find directory/ -name '*foo*' | head -1

Wishing programs do extra work desroys performance. Some programs run forever unless terminated. If you don't want a program to terminate early, don't send it a termination signal!

The the Unix philosophy is that users are smart and should be obeyed, not that programs should do what they think should have asked for instead. Forcing the user to negotiate with the program like a TARDIS is madness

rm doesn't print anything on stdout.

"rm -v" does.

Reply on news.ycombinator.com