Hacker News Reader

Comment by mvdtnz | Hacker News Reader

Just another reminder of the absolutely terrible state of our industry. Imagine anyone in any other industry taking this ridiculous stand. "Oh we don't fit roofs on Fridays, in case our roof leaks and we need to bring the roofer back in on the weekend". "We don't do Friday wisdom tooth removals, would hate for the dentist to have to fix a problem in the weekend". "No we don't fit tyres on Fridays, our tyre guy doesn't want to fix your tyre problem on the weekend".

The problem is the bugs, not the Friday releases. This industry is so utterly depressing to me. We get paid like kings to deliver like monkeys. Clown industry.

View on HN

The consequence of a bad deploy can be millions of dollars for a client or business. Depending on the industry, it can be a million dollars in the course of an hour. If a dev team works Monday to Friday, then it's likely they're going to be harder to reach and coordinate with during the weekend. It usually costs less money to wait an additional business day than to get a team of engineers in on a weekend to try and fix a problem in production.

If a roof leaks on a Sunday and the roofer doesn't work Sundays it's going to be similarly difficult to get the roofer to come and fix it. The difference is the order of magnitude of money lost for a leaky roof vs a software fuckup at a pharma company or a business conglomerate.

It's ultimately the responsibility of the developers to test their code sufficiently so that there never is a weekend problem to begin with, but the reality is there will always be bugs that crop up in production, and you can only control that so much. You can usually control deploying on a Friday though.

I really don't agree with any of your examples here, or the combativeness of your comment.

Roof work is usually a fair bit more deterministic than software.

For car work, it's not uncommon where I am for the tire guy, the vehicle inspector guy, or the guy who knows how to work on my brand of car to have unavailability.

My dentist has scheduled work earlier in the week with the explicit intent that if something goes wrong, I can be seen quicker. I appreciated this gesture, even if it wasn't exercised.

Roofing isn’t unskilled, but building codes make it sort of a commodity. If you just invented a new type of roof, I would not want you to install the very first one on Friday night and then send home every available expert to party or travel or whatever for the next two days.

The core of your argument is that it's the worker deciding this, and that they don't want to fix issues over the weekend.

That's not true.

It's the business deciding that they don't want to ship bugs on Fridays and then take the hit for customers having a bad experience over the weekend if the issue can't be resolved. What if you let someone deploy on a Fri and then they go skiing for the weekend? Sure, someone else can step in, but maybe it takes them an extra 2-3 hours to solve the problem. That might cost you a renewal.

Indeed, it's the business making the (IMHO poor) decision that the best way to address this issue is more manual oversight, and preventing release at certain times.

There are often other, better ways to address the underlying issue; "no releases on Friday" is a band-aid fix.

First off, we're still saying we deploy 16 days out of the month. I'm sorry that's not good enough for you? Do you want cookies from management for deploying every single day and on christmas eve? We'll host an extra pizza party in your honor (no bonuses this year, even though you did 33 deploys on Fridays).

Not to mention I worked at places that deployed twice a month, or deployed once a month, only with CTO intervention, which took four+ hours and multiple devops and engineering people doing manual shit, even with Kubernetes etc. So individual engineers deploying by themselves 16 days out of the month is "manual oversight" and "poor business decision". what a joke.

Also, what, a band-aid fix? I wouldn't work at a place that made me deploy major stuff on Fridays, first off. I have a life, so I'll take my bandaids, thanks. You think because people write tests and follow a spec that you can deploy MyBigFeature on a Friday without issue? absolutely not.

There are befits to small batches, short cycle time and completing at will.

They are nothing do do with what management wants or rewards.

I'm sorry to hear that you have only worked at places that are unaware of or unwilling to do good practices.

Your commentary on the whole is defensive, ignorant, rambling, ill-tempered and borderline insulting. It lacks the basic respect necessary to engage meaningfully, rather is more likely to create conflict. No further response is needed, thanks.

> There are befits to small batches, short cycle time and completing at will.

That has nothing to do with deployments on Fridays.

> you have only worked at places that are unaware of or unwilling to do good practices

I didn't say that. I also worked at places that did small deploys daily. But we still didn't deploy on Fridays. I deploy with my own projects, to production, many times each day. But I still try to avoid Fridays.

> No further response is needed, thanks.

I disagree. You complain about the industry as a whole but can't deal with basic criticism to back it up.

> Just another reminder of the absolutely terrible state of our industry. Imagine anyone in any other industry taking this ridiculous stand. "Oh we don't fit roofs on Fridays, in case our roof leaks and we need to bring the roofer back in on the weekend"

Agreed, the industry operates on a much more janky level than many other industries, but this syndrome is by no means limited to the field of software development.

If your roof is functioning perfectly well, fixing a non-critical problem on a Friday is less safe than fixing it on a weekday because if something goes wrong the next day, it's better that the "next day" be a day when the roofers are available. All other things being equal, I'd say fix your roof on a Monday so you have the next four days to call the roofers back if things go wrong.

It's the same reason you're safer booking an early flight, or scheduling surgery early in the day: if something goes wrong, at least you will get bumped to later in the day. If your originally scheduled flight or surgery was later, and something goes wrong, you will likely get bumped to the next day.

Obviously this rule doesn't apply to emergency fixes: if your roof has a bad leak, and it's a Friday, and you know it's going to rain on Saturday, you're still better off having it fixed on Friday.

Actually what's happening in those examples is that issues DO happen on the weekend, the customer just suffers or calls someone else to patch things up till it can be addressed correctly. Difference with tech work is that your employer typically suffers.

The thing is that the one roof doesn’t affect all your entire customer base. That isn’t quite a fair comparison to make due to the nature of software scale, but I can relate to the sentiment.

I don’t think this take is complete unless you include complexity and the risk of unintended consequences. We pretty much understand roofing, but software doesn’t struggle with the chains of the physical world and the base case can be significantly more complex.

Maybe it's just me, but if I knew the dentist will be away tomorrow, there's no way I am going to get my wisdom tooth removed today.

It depends on the type of software. A B2C SaaS that has continuous small releases with easy incremental deployments and rollbacks is incomparable to a bespoke B2B release that integrates with dozens of the customer's other systems.

The latter are typically deployed over the weekend with multiple companies/teams on standby as unexpected interactions are at a certain point of complexity a given.

Your quality aim is for instrumentation and verification to rapidly detect and access and triage issues since assuming you can hit 'deploy' and walk away because your code is perfect (delusional) ignores that 95% of the systems you integrate with aren't even under your control, and 'it worked in test and was signed off' is not an acceptable answer for a client's production system being down on Monday.

Speak for yourself. In my shop we have CI/CD and we ship whenever we're ready to ship regardless of the day or time.

Reply on news.ycombinator.com