Hacker News Reader

Comment by fleaflicker | Hacker News Reader

I asked how to roll back the changes. He said "we don't roll back, we fix the code."

Not the best idea. I don't debug as well under the extreme pressure of my site being broken with the clock ticking.

It's not always like that. John Allspaw's answer is correct in the sense that actual rollbacks aren't done -- the fix will be a new push -- but sometimes, before you do that, you will just turn the new feature off in production.

This is possible because many new behaviour is controlled by "feature flags" which are associated with the server configuration. So you have the benefit of everybody, developers and production, staying very close to trunk, but still being able to have radically different behavior on development machines versus production.

That said, I have participated in debugging sessions on a product which used continuous deployment, and they can indeed be nerve-wracking. Personally I wouldn't want to use CD by itself, without a good culture of code review and a great test suite.

Feature flags are a very powerful feature, especially for services. Many of the most successful web companies (amazon, google, facebook, twitter) have entire frameworks and infrastructure dedicated to supporting the design pattern.

You'd probably find it easier when the deployed changes were extremely small. Our last automated deploy was a single line change. Most are bigger, but not huge.

Also, while they don't do full rollbacks, I suspect more than one fix has been "remove the offending code until we can figure out what's wrong".

Agree, it seems to me that an emphasis is on time production deployment takes rather than rollback vs fix

I've seen single line changes cause data loss, corruption, system outages, remote root exploits...

I'm not sure that the number of lines makes the change more or less risky.

And how exactly does this relate to the way they deploy their code? As far as I can tell, they actually review code before marking it ready for deploy. That kind of changes would be an issue with "usual" large batch deploys as well.

I've also seen many devs look at a broken release and instantly realize their mistake. A forgotten production config, a hard coded variable, or an empty cache. If code is in fact reviewed before going to production, the risk is significantly lower.

Nice choice of username, by the way.

The safest change to make to a stable system is the smallest change possible.

When you know you can't really rollback, I'm sure you're helluva lot more sure that your code won't take down the site

I'd bet it's semantics.

Rolling back the code-base would mean everyone else would need to roll back theirs, and you'd lose history. It's easier across the board to simply re-write what it was - a "roll forward", if you will.

Not the best idea, if you don't debug as well under such pressure. Some people do.

Or, more commonly, people think they don't perform well under this kind of pressure, but they actually do. Usual scenario:

Employee calls the CEO of the company in the middle of the night panicked, saying the site is down and there is no way they can fix the problem, they don't even have the right skill set.

CEO tells the employee to calm down and do their best.

Site is back online twelve minutes later.

More likely, the developer can handle the tech pressure, but having his/her manager FREAKING OUT and over his shoulder asking questions the whole time is what really causes the slowdown.

Step 0. in getting started with continuous deployment is having an organization that doesn't lose it's mind every time there's a blip.

That often means you need managers who are especially good at (or at least dedicated to) deflecting the freak out/crazy.

CD won't increase your changed related incidents, but to paraphrase the IBM parable, "no one ever got fired, for doing quarterly releases, and heavy QA."

I used to code for a website handling about 25,000 logged in users per day, and there was a post-commit hook on SVN which pushed changes live, immediately. Working on that taught me a lot (mostly how not* to do things), one thing I learnt is that I code much, much faster under that kind of pressure.

* Note: I am not endorsing this as a setup, it's crazy. I don't think I've ever seen such high staff turnover.

Reply on news.ycombinator.com