Signals in Prod: Dangers and Pitfalls | Hacker News Reader

Signals in Prod: Dangers and Pitfalls

developers.facebook.com

50 pointsmooreds16 comments

Good writeup, and lots of detail.

But my immediate conclusion isn't "Signals are dangerous and let's go deep on the complexity thereof". Instead, my primary conclusion really is "Someone forgot to write a testcase to make sure log rotation behavior is covered, so of course it might regress".

Both are true.

I would throw in a third conclusion: given that you're directly using signals (without a wrapper to rationalize the interface) your team should be composed of people who fully understand how signals work. Neither the person who wrote this code or the person who reviewed it had internalized the fact that "the default SIGHUP handler kills the process", and therefore using signals was an inappropriate decision for this team.

But this is an exceptionally high bar; most interfaces do not require that your devs have memorized so many details. Signals expose a terrible interface and there is essentially no satisfying way to make use of them. We should expect better from the tools we use.

I dunno, I don't even program in C and when scrolling through the diff I immediately said to myself, "Hey, I bet that callback registration they removed for SIGHUP could be an issue; I would have no-op'ed the callback and left it in there." Of course, I had the privilege of knowing that there was something wrong with that specific code diff relating to signals.

My take is similar to yours: only use tech and features the plurality your team understands. If the people leave that are the only ones who understand the fancy stuff, then you run into this type of issue. At my current job, the guy who setup the database put all kinds of fancy extensions, triggers, and procedures in there and no one knows how to dissect it or what it all does, so now we're running old versions of code in a database container. I am sure someday it will be my job to replace that black box with something else, and probably just move the data from one to the other with ETL methods.

"At first, nothing had gone awry, but at midnight the next day, everything mysteriously started falling over."

Everyone has a test environment and some people have a prod environment too.

If any shop lets their code changes marinate for 24 hours in a test environment then power to you, you have more patience than any developer I've ever worked with. It feels like a uniquely ops-y mentality to roll out changes on the order of days or weeks.

If that change affect something that only occurs once a day, you have no choice but to wait no?

Oh get off my lawn, kid. It's not signals that are dangerous; it's removing code that's dangerous. You thought it was redundant, but it turned out you were wrong.

FWIW, if you can get away with it, it's nice to have your log writer check if the file has been rotated when writing if it's been 60 seconds or more since the last check; this allows you to avoid signalling, although you do need to wait to compress the logs, if log compression is part of your rotation.

This was a great writeup. I have read a few great low level articles from facebook and it definetly brings up my view of fb. I hope they continue to produce quality content like this.

Completely unrelated, but useful: POSIX kill (or kill()) with signal 0 is a portable way to check if a process controllable by the current user is alive.

Could you elaborate on what makes that a great way to check and what advantage that has over other methods?

It doesn’t do anything but returns a result based on the existence of the process and your ability to signal it, which makes it a sort of “dry run” of sorts. Note that such a check is inherently racy, though.

> Note that such a check is inherently racy, though.

Do you mean in a kind of "time of check vs. time of use" kind of way?

I would suppose that if the answer is negative that would at least stay true no?

Yes. If the process exits (and is perhaps replaced by something else!) in between the two checks you will take actions based on stale information.

I can't argue with any of this, but like xkcds "Teach wearing a condom"[1], everything points to something having gone horribly wrong to have this lesson.

[1] https://xkcd.com/463/

> High Complexity in Multithreaded Environments

This is pretty much inherent in multithreaded environments.

Reply on news.ycombinator.com