Hot Takes on Code Freezes | Hacker News Reader

Hot Takes on Code Freezes

jeli.io

47 pointskiyanwang33 comments

TL;DR – Incidents can still happen despite code freezes. Developers on call should be paid extra. Account for the holiday season during quarterly planning. Time during deploy freezes can be spent on other tasks like planning, specs, tooling, testing.

Are any of these really "hot takes"?

> Developers on call should be paid extra

Sadly there are many organisations out there who don't respect employee annual leave and expect their engineering team to be on-call without compensation. The usual excuses are things like "it's the holidays, it'll be quiet".

GTFO of those shops if you can as a practitioner. Don’t take workplace abuse or provide uncompensated time.

One of the better "hot takes" was saved till the end:

>A deploy freeze means that folks are not actively trying to make progress on a dev project that needs to be deployed. Deploy freezes need to equate to either time off, or time spent on things that do not need to be deployed when complete.

The author does a bad job explaining why. If you allow deployable work to progress without deployment during a holiday code freeze, you are just shifting, sometimes magnifying risk post-code freeze. All that undeployed work during the code freezer will increase the rate of change tremendously post-code-freeze.

Deploy freezes are not necessarily about preventing incidents, but rather controlling the risk (which is not just "what broke" but "how long was it broken"). A normally small incident that happens while everyone is out enjoying their holidays might end up being worse than a large incident that happens at 10am on a Tuesday while everyone is at their desks and can respond immediately.

There's also varying costs (not all monetary) to responding to an incident. I'm going to be very upset at my employer if I get called in on Christmas because there was a deploy going on that caused issues. First week of January, that's just expected.

This. The goal is to push the looming incident back until your staff will be on the grid, awake, and sober, and your customers are just having a normal day.

If we freeze for two weeks, spending the next four weeks deploying six weeks’ worth of feature branches shouldn’t be that big a deal. It’s another form of tech debt, and your process needs to allow using it judiciously (CI/CD often gets this wrong).

They are hot/new if one is fresh to the game. If you've been around like 10 years be prepared for many repeating patterns. My personal favourite is the flapping between Server side and Client side business-logic - and the same circular conversations.

I think the hot take is that although these are best practices, not many companies actually do them.

Reading posts like this one, I am very glad that I work at a place where we have

1) an engineer on-call rotation for every discipline - no database engineer has to troubleshoot a broken CSS rule

2) on-call engineers get paid a big chunk of change, and even more on holidays

3) all the on-call engineers are much smarter than I am (a little harder to replicate this one, unless you're dumb like me)

I honestly kind of forget that it even exists because it runs so smoothly. Good opportunity for me to get some perspective and be grateful!

Edit: since someone asked, here are some other cool things that my engineering org does so smoothly that I forget they exist

- Very very good automated build monitoring: when you push code for review / to internal / to prod, you know what happens to it as it happens.

- Business tech is something I've literally never had to think about for more than eight seconds – laptop, idm, badge, slack, outlook, etc has worked absolutely perfectly the entire time I've been here (except, weirdly, this one meeting room that refuses to stay reserved.)

- Testing: we have a ridiculously competent QA apparatus. Our product is used by customers with extremely stringent UI requirements, and so regression testing is mission-critical, even for tiny UI bugs. Every team has a QA, but we also have a QA Architect position who helps guide org-wide decisions. Generally, QAs are also very respected and treated as peer engineers by feature developers, and I think it shows.

I work at a big SaaS company, and there are entire teams of people devoted to developer tools, so this stuff is all """basic""", I would argue; but man, it's easy to forget how much human effort has to go into making it basic, and how often we fuck up the "easy" stuff.

Where do you work?

I try to keep my work identity a little distant from my online presence, but feel free to hit me at the email on my HN profile if you're really curious. You could probably track it down if you really cared to.

It's an excellent place to work, particularly as an engineer, and we're always hiring.

I don't care where you work but would love if you could point/list some other smart things they do. Those things where it's so smooth nobody notices - that is the hot shit to copy.

I just added a couple to my top comment.

> Hot Take: Putting a deploy freeze in place does not mean you will be incident free.

I find this does not match my experience at all! The number one cause of incidents are code or configuration changes. Freeze those and the incidents DO go away 95%. Of course you need to update eventually and the problems will begin again, but freezing when there is the least amount of staff to fight fires is very sensible???

of course, nothing is 100% but let's not rock the boat so that the rare traffic based incident can be dealt with by the few staff that are there.

Anecdotally, configuration changes seem to cause the biggest outages these days.

We call them change freezes, they are generally done when there are less people around. This allows employees to celebrate holidays.

These do reduce, but don't elimate incidents.

Unilateral change blocks, stink, but don't put support in the weird position of taking on something they can't.

Also, in some climates, weather will play a role in IT activities. Even those working from home are not immune from weather related events that might affect their house / apartment. Pipes freezing will end most IT related functions from that employee. People who might have to work regardless of code freeze might have other issues.

The air conditioning in the server room failing when it is -24F is a possibility.

One company that employed me has ops based in the Philippines. Doesn't matter how much you pay, if there's a typhoon the grid is going down and ops will be unavailable.

I wonder how many of us have vendors that depend on support people in the Ukraine. I suppose that is an unexpected and unwelcome risk.

Funnily enough the same company employ (via agency) devs in the UKR. Most of the country is stable.

It is better to just accept that nothing is going to happen between Thanksgiving and New Years. Chill out, enjoy the holidays, and don't push so hard.

I don't think "preventing issues" is the reason for deploy freezes. It's more about minimizing change with the hope of minimizing the surface area the skeleton crew needs to consider when an issue does happen.

I personally find them frustrating because they throw off our regular release cycle. Sure, we didn't deploy any potentially buggy work (or any fixes for known bugs...), but now we have to manage a giant release when we 'thaw'. And make sure no work gets left out, which is more likely since we're off our normal cadence.

Another option is (if possible) not to have a release cadence in the first place. If deployments are happening regularly everyday as a part of small PRs, you just get a backlog of code reviews. Sure some releases will require orchestration where order is important but better than one giant release.

DB migrations are a different story imo, and should get more planning. I’m guessing there would be other stacks/domains where regular release intervals suit customers better though.

"Deploy freezes shouldn’t mean idle engineers."

Why not? Thats the best part.

If you have nothing to do at work during the holidays then take the time off and actually enjoy it. Sitting in front of your laptop on the clock doing nothing is pointless for everyone.

Getting paid while having low expectations sounds wonderful, actually. Perfect time to work on pet peeves or professional devlopment.

Never allowed to have a “slow day”, gotta be grinding on at 100% effort all the time! Money to be made!

Fitter.. Healthier.. More Productive..

> Everyone who is on-call or working support during the holidays needs to be given mandatory additional time off and should receive holiday pay.

Companies don't want to do that. That costs money. So instead, they developed the idea of a "code freeze" which doesn't cost money (directly). In the mind of management, if there aren't any code changes, then nothing can break. Obviously, this is not the case, but that's not the point. They're not going to pay more for overtime and the marginal cost of deploying changes over the holidays does not justify the expense.

I worked at an ecom unicorn that did this and had been doing so for about a decade. I think it saved them quite a bit of money by giving extra time off to ops in exchange for their extra work during the holidays.

Code freezes (in my experience) don't actually reduce the number of bugs or incidents. I think in all probability they increase the number of incidents and cost money because of the warm up period.

But they do shift the timing when incidents occur, and that can be nice if your customers are sensitive to holiday sales. It's also nice to your employees. Paying your SREs a bit more for their tour of duty to the customers and employees is worth it.

Incidents are mostly caused by untested configuration changes, infrastructure failures and often known limitations of software. A “hot take” could be something like code freezes are bad cause when it comes to most software and incidents the risk with changing the ways of working and not improving the software can be bigger than the risk with new releases.

I think it is ok with code freezes though.

Reply on news.ycombinator.com