Hacker News Reader

Reminds me of this old classic:

Openoffice can’t print on Tuesdays

https://news.ycombinator.com/item?id=8171956

Reminds me of the build script I found in a single-dev project I inherited: failed due to truncated/missing leading zero when the wall clock was earlier than 10 am. My predecessor was notoriously late at the office, it would have worked for him every single time.

Ok, let me up the game. I had one simple short-lived cron command that failed on even minutes before h:20 where 5AM < h < 8AM on Sat, Sun, Mon.

Cause was another job that hammered IO because reasons, so mine was being killed by some unreported IO timeout after 1min, and retried the next minute (so every other minute). The other one was scheduled to run on off business time (weekends and Monday before people get to work) and had some burstiness that made the IO go down enough every other minute which allowed my short-execution one to proceed.

top was silent because IO, took us a long time to even thing about iotop because the failure was non obviously tied to IO.

Reply on news.ycombinator.com