Hacker News Reader

Comment by alfalfasprout | Hacker News Reader

The best one is companywide code freeze well before the actual holidays (minimum 3-4 days before). That gives you 3-4 days to discover a whole slew of possibly nasty things and revert them before the week happens.

This keeps the likelihood of anything actually bad happening during the holidays way, way down.

View on HN

This. A large fraction of outages are an unintended result of a planned change, so it's wise to freeze those on weeks when no one is around and sometimes other critical times (like during Black Friday sales for a shopping site). If you do this, holiday weeks are usually super quiet.

My worst holiday oncall shift as a SRE was when a couple SWEs on a semi-related team didn't honor any of our best practices, instead rolling out an unannounced change during the production freeze with no premortem or rollback plan. (This was 12 years ago, and clearly the larger org had failed in communicating cultural expectations to everyone.)

I would go so far as to say that the vast majority of outages are caused directly by code changes. If your org has a lot of commit churn, you’ll find a few more bugs in the calm of a holiday change freeze, as hundreds of patches take their first journey past 72 hours of uptime.

However... it can also trigger new behaviour because programs / services run for longer than normal because deployments aren't being refreshed as frequently as normal.

This seems like it's just being contrarian for the point of it, even if it doesn't lead to any interesting conversation.

I get it, we're hackers, we like to think of edge cases and are pedantic as hell. But pedantry and edge cases aren't always useful in conversation.

I don't think it's even that much of an edge case. We've definitely had memory leaks on all sorts of services just because they were running longer than usual without a deployment.

This caused at least one outage in my career. Memory leak caused every single app server to crash in the space of a few hours on new years eve; regular deploys had been hiding it.

This is unfortunately all too common, but it can at least be solved with a simple restart - and truly fixed after the holidays, which is much better than a 4h debug fest because of a typo in some code someone wanted to quickly get in before the holidays.

We had a problem where HikariCP was slowly recycling db connections but failing to open new ones. Guess what happened when the app ran long enough the final conn in the pool closed...

I speak from experience, unfortunately :)

exactly this, in companies that had some kind of moratorium during holidays the on-call was often the quietest time. Majority outages are triggered by code changes, even those that seemingly are unrelated.

Reply on news.ycombinator.com