Comment by akbar501 | Hacker News Reader

Comment by akbar501 | Hacker News Reader

Parent Full thread

Great post. I could not agree more. To further the article, CI/CD are often neglected by small teams / startups. However, startups would benefit more than large teams (as a percent of time) by automating the build and deploy processes.

The only point I've heard a counter to is "Automate Rollback". Noah Sussman (previously of Etsy) mentioned that they never rolled back (at a Groupinar). When a bug was pushed to production, they would fix in dev, test, and deploy. If I remember correctly, he mentioned their time to fix was < 7 mins (or better).

Also, when I first started in my career, one of our most senior admins stated something very similar. In a meeting, I suggested we have a rollback mechanism. The response was that code is like a river, keep it moving in one direction.

In practice, I've found that rollbacks should be reserved for catastrophic events where you are literally running your deployment software to reset infrastructure, dependencies and application code.

Came here to say the same thing. You can't roll back without rolling back your database, but I've yet to see a good way to accomplish that. What happens to the data you collected since your last backup? How much downtime do you need for the db rollback?

Also, when you do your rollback, what happens to?:

- data in caches like memcached or redis

- images on S3/etc whose URLs are stored in the database

- session cookies on users' browsers

- state you've created on systems you integrate with, e.g. fulfillment, warehouses, email lists, . . .

The database is only a portion of your app's "state", and if you suddenly wrench it out of sync with the other systems, you're going to cause problems.

If you absolutely need to stop the world while you develop & deploy a fix, then have an "unavailable" message you can quickly drop in for a few minutes.

Maybe you'll find a bug that really does require restoring to a backup, but that should be rare and will cause a lot of pain, so making this automated seems very scary. You want to do your restore in a controlled, considered way, so you don't cause even more data mangling than you've already got.

Agreed, we've never done a rollback as well, but having the automation in place makes it easy if necessary. Automating still should be triggered by a manual process.

Reply on news.ycombinator.com