Hacker News Reader

Comment by exdsq | Hacker News Reader

I know this can be a good idea because it helps you catch stuff, but man how could you convince patients that you're a reliable company if you did lose something by deliberately turning stuff off and on again. I wonder if you could argue it's negligence. Every day seems like you're just inviting a mistake at some point!

View on HN

Doesn't routine make you better at things?

(And since everybody is now going to share war stories... When I worked at Con-way, now XPO Logistics, they had the pop-tart incident, where a kitchen mishap triggered fire regulations that mandated the data center be taken down, and since they had absolutely *zero* routine in restarting their data center, folks got to debug the backup plans in real-time, until two weeks later all service had been restored. Kind of. Me suggesting we should test this more often was only perceived as a joke in bad taste. I thought it was a swell idea, since our Sundays were scheduled downtime anyway. Yep, the whole day. Every one or two weeks. Suffice to say, it wasn't a very long stint.)

People improve with routine when the environment is static, or within certain ranges. Routine degrades performance when previous assumptions shift (or are changed by dynamic agents).

The gamble is to have small, contained issues you can deal with in a timely manner, vs full scale propagated failures you'd have to deal with at the worse time ever.

It's like accidents during fire drills, they happen, yet it's worth doing all things considered.

That's the whole point. Every feature they build absolutely has to take failure into account so they do. Whereas if you only failed in emergencies, you would find stuff that didn't handle the failure well.

Sure I get this in principle but if you keep rebooting something can you instigate errors that are very unlikely to normally occur, by increasing the probability? Feels like you're going to trip an edge case maybe even lower in the hardware stack than your software if you do this so often.

I mean restarting hardware once a day isn't going to hurt the hardware (indeed its the norm in some places to shut down many racks after business hours), and software will probably only see less bugs.

You could say the same thing about never testing your DR/backup capabilities. That's way more negligent.

You never want to use the parachute, but should always know it works.

Reply on news.ycombinator.com