Hacker News Reader

Comment by Sebb767 | Hacker News Reader

> A lazy programmer do not deploy in production, they instruct Jenkins to do that. Therefore a lazy programmer is not afraid of deploying on Friday afternoon.

Just because Jenkins does the final "cp && restart" does not mean that critical bug will not ruin your weekend. The tests mentioned before help, but the unexpected is, per definition, unexpected.

In fact, when the house is burning, I found CI deploys to be quite a slowdown. But this might just be me.

View on HN

Yeah, not deploying on a Friday has nothing to do with a lack of CI; it’s related to the human culture of having a weekend, which reduces an organisation’s ability to handle the unforeseen consequences of pushing to production.

“So just roll back” I hear you say, to which I respond: “it ain’t always that easy”. You can’t always just turn the reactor off once the meltdown is in progress.

We've had this issue many times with dependency management.

CI will rebuild after you revert everything and still fail to deploy if you pushed a versioned library from your broken code.

Go into your environment and manually roll back to a working image... and learn a powerful lesson.

I'm not sure what you're saying here. I can see this being a problem when you have cross-team dependencies and your roll-back requires coordination.

Even then it's manageable if your production deployments are version pinned, it only becomes a problem if schemas in your database or messaging have had non compatible changes and data has been written (i.e you have binary and data dependencies in a release)

All this being said with the assumption that the CI/CD pipeline has been designed to best practices.

OP here: I agree on some level. I have perfected my vim skills by actually modifying the production php source code in my youth. However as things get more and more complex, it helps to script them down so that you don't have to remember a set of commands in a given order. I see the deploy script in this case as a "documentation for lazy" :-).

Fair enough. Our CI does build+deploy and we're on a bit of a budget, so the process tends to be `git push`, wait 5-10 min for the pipeline to complete, check if it worked, repeat otherwise. But that's really something to optimize in our infrastructure; in an ideal case, I'm on your side :)

Yes. I say deploy in friday, but you better call it out explicitely. We’d call it cowboy friday anytime someone suggested deploying on friday. That person immediately volunteered to be on call. And if anything went south, they lost social capital and had to eat crow (mostly playfully, more seriously if it became a pattern). If enough people agreed the risk was worth the reward, and at least one QA was willing to vouch for the push, the yeah, push as long as there’s enough time to rollback before people leave for the day.

Mostly, it was easier to not push on friday.

> I found CI deploys to be quite a slowdown. But this might just be me.

It depends how you think about it I think. Meaning, it'll feel like quite a slowdown but exists to help mitigate errors made by humans experiencing varying degrees of panic while their house is burning. In my opinion CI deploys are meant to provide consistency, not speed (although people generally experience speed ups in their deployment process when there are no longer "works on my machine" type issues that pop up each deploy causing slowdowns in new and exciting ways).

But I also think that programmers flaunt "Friday afternoon deploys" too much in either direction. It shouldn't be the end of the world but it also is just an unnecessary risk if you can help it.

Well, I guess the point here is to rollback to the previous version and find and fix the bug on Monday.

Sometimes, yes. But the bug might not immediately be obvious (for example a memory leak) or cause data corruption, in which case you need to get a backup - and you probably don't use a backup without manual approval.

A good infrastructure will help you at every point, but it will still take time.

I'm curious how many companies have a working rollback. I think the majority probably have to keep patching until they get it right or get it stable.

maybe because by definition you lose data on rollbacks? You can re-run all events/messages (if your company architecture supports it) but that gets more complicated, usually patching is easier and faster. Also, downtime.

Deploying often means a lot more than just “changing some binary somewhere in prod” too.

Apple won’t help you during weekends. Our experience. There are dependencies, you at first don’t think of.

Why is a developer deploying to production in the first place? To minimize the work for everyone the code should have gone through QA before it was passed to ops. Ops should be confident that QA did their job sufficiently (“A lazy programmer knows how to delegate”) and shouldn’t be afraid to start the deploy job whenever.

Not all companies/teams have dedicated QA and Ops teams.

In the (very large) company I work at, the engineers are responsible for writing the code, testing it, and supporting it once it's live. That includes being Oncall and being paged if something goes wrong.

For a small company with a fairly stable feature release schedule, that is fine. But in a large setting where expectations for new features is high, that seems like a scam. That is it looks like they are trying to get the developer to work double (or triple) shifts but only pay for one.

Why are there separate QA and Ops teams?

Ops usually understand infrastructure, and QA usually not so much. The overall mindset is a little bit different, i.e. QA tries hard to break it, and Ops tries hard to keep it running. And the required skills might also be different (distinct sets of tools, techniques, certifications, etc.)

But I can see that it might be useful to have it all together... (DevOps, QAOps)

Dude, I'm so jealous that you haven't even heard of DevOps. I don't know where you must work but please give me a job.

The trouble seems to be that "DevOps" is the new "Agile". How often is it just an excuse to dump non-development responsibilities on the dev team, and to ignore boring but time-proven processes and tools that specialists might have preferred if you'd hired them for those responsibilities instead?

Reply on news.ycombinator.com