Hacker News Reader

Comment by imdsm | Hacker News Reader

> otherwise your 3 hour code review, build, q/a, deploy pipeline will cost the company $180,000

Taking code review away, because we all know we're not going to sit around waiting for a review while a patch needs to go out urgently, if your build, test, and deploy pipelines take three hours then you have some serious problems that you need to address, and containers aren't it.

There are methods for handling hotfixes/patches to production quickly that work well in high volume sales website setups.

View on HN

> Taking code review away, because we all know we're not going to sit around waiting for a review while a patch needs to go out urgently

I worked at a trading firm where you could add a zero to that cost for a 3-hour outage, and I can tell you that the one thing we would absolutely never skimp on was code review. Because the cost of a bad "fix" that actually makes things worse has the potential to be greater still, and because humans are most likely to make silly mistakes when they're working under intense pressure.

What we would do instead is slightly intensify the code review process by pair programming the hotfix, and ensuring that a third developer who was familiar with the system in question was standing by to follow up with an immediate review.

I really like that approach. It also takes pressure off the tech lead or whoever is implementing the fix, and transforms the patch into a full-team responsibility. I bet this sort of behavior makes for strong and effective teams.

Pair programming anything as urgent as a hotfix works really well. It takes some pressure off of the developer working on it and turns it into a team event.

We will even sometimes keep the video call open until deployment is done and production validation is complete - the devops guys get the information they need, someone else is keeping an eye on the checklist and calling out items if necessary, etc.

Oh yeah, absolutely. I was happiest when I also had the attending ops person looking over my shoulder while I worked on the fix.

Please do tell. Our containers never take less than 10 minutes to deploy (a single one).

Well 10 minutes is way less than 3 hours.

Certainly. But it does add up and it does kill motivation for rapid iteration. :(

and then we're back to the original statement, if you're rapidly iterating hot fixes you have serious problems and likely doing it wrong.

10 minutes to deploy to production should be fine, but rapid iteration shouldn't be happening in production. It sounds more like the complaint is related to running a similar setup in dev, and taking 10 minutes to see changes during development (which is understandably too long).

Yep, that's what I meant. I have no choice but to run a local k8s cluster or else I can't test.

I have app stacks that deploy a dozen containers in seconds because they are stateless and close to "functional" – just transformations over inputs.

I have app stacks that deploy a dozen containers over an hour because the orchestration takes time: signal the old containers to drain, pause for an app with a very long initialization time to settle, gradually roll traffic to the new one to let caches warm, and then repeat.

In both of these cases, deployment is a function of the application. There's nothing infrastructural that puts a time floor on things.

Sure, not contending that. It's just that in my memories blue/green deployments still took less time, although I can't say how much.

Not a serious reply, but it may interesting: One million containers over 3500 nodes in 2 minutes: https://channel9.msdn.com/Events/Ignite/Microsoft-Ignite-Orl...

Reply on news.ycombinator.com