Hacker News Reader

Comment by synicalx | Hacker News Reader

Good on them for doing a relatively detailed, honest and public review like this. But goddamn that recovery time is absolutely unacceptable for a modern mission-critical product like this, especially since they've essentially forced everyone onto their cloud platform. Not having the ability to automatically restore backups in 2022, when you're a MAJOR SaaS provider that stores massive amounts of essential data is just mind boggling to me.

Hopefully this event has enough of an impact for them internally (and financially) that they seriously reevaluate things and maybe invest in some SRE capability, and better tooling.

View on HN

I think what they're implying is that they can restore _everyone's_ data quickly (i.e. 'usual' infra failure), and one Jira instance's data fairly quickly (but some manual steps), but the problem is they nuked a large subset of customers while everything else kept running. This means that they can't do a complete automatic restore, and instead have to deal with it customer at a time.

I'm sure that they're going to make their customer at a time thing properly automatic now... Still a big failure, but a little more subtle than you make out.

(disclaimer - worked for Atlassian years ago)

The thing is that the data was wiped as part of a "Compliance Delete", albeit a wrongly executed one.

I am actually more surprised that they are able restore the data at all. If you can recover from a Compliance Delete, is it really a Compliance Delete?

If immutable backups can be nuked by a single compliance delete request why bother making backups at all? All it takes is a delete request per day to get rid of them.

By removing data and only keeping a 30 day backup any compliance deletes still simply take ~a month to complete.

I think most teams delete the live data from the database then wait for the backups to expire after N days.

It's clearly not mission critical, Atlassian has just demonstrated that. "Move fast and break things" and "mission critical" are opposing forces that just do not mesh.

My team and I run emergency services communications infrastructure with 99.999% availability requirements and our company has to pay an abatement if we don't maintain those levels. That's actually mission critical. We don't run ANY script or SQL command in production that deletes data without having tested it in our dev/test environment, documented the steps of the change and the results of testing, having the change reviewed internally, and approved by the government.

We can get these changes approved and into production in around a week normally. "Move fast and break things" is great, until the wrong thing breaks.

The list of affected data that was wiped out by the compliance delete and now requires recovery also includes the metadata in their orchestration system, and what sounds like other multi-tenant data stores. It's not just restoring a single, entire PostgreSQL database from backups.

Lesson learned is not not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong inputs.

I agree that the timeline is not really acceptable. I am not sure anything mission critical should be hinged off a ticketing system, though. At best, things should be reported to the ticketing system, but never triggered from it as some sort of source of truth.

Reply on news.ycombinator.com