Comment by ghshephard | Hacker News Reader

Comment by ghshephard | Hacker News Reader

I think the problem is in our definition of "Mission Critical" which is a fuzzy word. Depending on how "critical" a failure would be, continuous deployment may or may not make sense.

The author talk about 2-year certification periods as though that were a big deal, which indicates he has a different definition of "Mission Critical" than, say, a Shuttle Engineer, Power Grid Manager, or Nuclear Regulatory Organization (All of which experience 5+ year certification periods _after_ the release of a new system)

A multi-hour failure at amazon.com has different consequences than an multi-hour failure of the west coast power grid.

As one who deals with organizations that have QA, Code Drop, Full Scale Test Environment _in addition_ to their staging environment prior to allowing anything (including software that has been fully regressed with the coverage described in the article) to be pushed into production, I understand the resistance to deploying "latest and greatest"

The workaround - maintain a Full Scale Test Environment (preferably, _multiple_ such environments) internally for your Engineering, QA and Operations Group - and drive your continuous deployment builds into _those_ environments, and then stage checkpoint releases for your "Production" environments.

The only downside is that production customers don't get the opportunity to provide rapid-feedback on new features, so you still get a bit of the negatives associated with waterfall class requirements-architecture-design-implmement lifecycles, but, you are able to enjoy the benefits of rapid prototyping and bug triage internally that are associated with continuous builds, massive automation frameworks, and close-to-real-world deployments of your software.

Continuous deployment is all about avoiding such failures. As such the more "critical" the system is the more valuable this form of deployment is (though I also believe its also valuable for less critical systems!). And of course, continuous deployment is not complete without shakedown and back-out procedures.

I don't buy that the more critical a system is, the more valuable continuous deployment is. The downside of continuous deployment is that you see more problems in production than you would with a fully regressed build. The advantage is that those problems cost 1/10th as much to fix, releases occur much more frequently, and you have a tight feedback chain between your customer and the product.

The argument for Continuous Deployment is that you _always_ see problems in production, and that all you are injecting from moving from a staged-release cycle to CD is that you are now seeing 10-15% more issues (Let's say 60 P2 or greater Issues instead of 50 P2 or greater Issues), but the cost of fixing everything has dropped significantly, fixes occur much, much more quickly (sometimes same day instead of multiple months), and you are able to enjoy the productivity advantages of your software that is finely tuned to the users needs much, much more quickly. CD can (and usually does) result in less downtime than stages releases - mostly because of the rapid cycle from coding -> error detection -> problem resolution.

The _only_ problem as I see it, is that you don't have as much control over the probability of a P1 issue hitting your system. That's fine in the case of something like Amazon.com, where a P1 issue might cost the organization $10 million dollars, but they've received $50 million dollars in value from using CD. It's not the case where a P1 issue might result in a catastrophic loss measured in 10s of Billions of dollars (Power Grid, Shuttle Launch, Nuclear Systems) - for those, you need to stage ensure 100% coverage/regression/code review. In fact, you need 100% coverage/review of your _development techniques_, not just the code produced.

I think another way of looking at this is what can you do to make critical systems more robust and resilient.

The capability to monitor, deploy, and rollback new features/fixes on short notice would seem to be a part of that, even if actually used infrequently. I would think that you would need to be able to respond to a change in the environment the system is operating in as much ensure that a new release doesn't contain bugs that would manifest in your current understanding of the operating environment.

I am suggesting that much if not all of the infrastructure needed to reliably patch/rollback critical systems can also be used for continuous deployment at the option of the development team and/or the customer.

So in the event that a P1 hit a continuously operating network application (e.g. power grid) the ability to deploy and rollback new features rapidly in response might be a valuable option to have. It's an approach that increases resilience. This does not mean you have to do this all of the time.

I absolutely agree with you that continuous deployment increases resilience. The problem is that it also increases instability.

Perhaps the closest example I can think of with regards to continuous deployment in mission critical situations are the martian rovers - I think they had some real-time deployment of new code. But, the implications of a problem with them were relatively minor - a few hundred million dollars, and no lives lost.

Are there any examples of continuous deployment in a scenario in which hundreds of lives and/or billions of dollars are at stake?

Reply on news.ycombinator.com