miketria

Hi Ranteki, you're right that the RTO for this incident is far longer than any of the ones listed on the doc I linked above. That's because our RPO/RTO targets are set at the service level and not at the level of a "customer". This is part of the problem and demonstrates a gap both in what the doc is meant to express and a gap in our automation. Both will be reviewed in the PIR. Also, the answer to (1) and (2) is yes.

Hi, this is Mike from Atlassian Engineering. Strongly agree with this. I'd say that if you can afford it, don't do the hard deletes on a schedule though. You never know when there's a system out there referring to soft deleted data that fails once the data is hard deleted. Hard deletes should feel frightening because they are frightening.

Hi, this is Mike from Atlassian Engineering. Not a minor issue. Once we knew the extent and severity of the incident, we had hundreds of engineers engaged and working to restore service.

Hi, this is Mike from Atlassian Engineering. You are right that the checks need to improve to reduce human error, but that's only half of it. I don't see this as human error though. It's a system error. We will be doing some work to make these kind of hard deletes impossible in our system.

Hi, this is Mike from Atlassian Engineering. For the customers impacted by this incident covered by an SLA, we will adhere to our contractual terms. However, given the long duration of this outage, we are planning to go above and beyond for our impacted customers. We are currently focused on restoring service, but after that will be discussing how we can make it right for each impacted customer.

Hi, this is Mike from Atlassian Engineering. You are right the communications from us have not lived up to our standard. We will focus on this specifically once we restore service and get the post incident review out there. More details here: https://www.atlassian.com/engineering/april-2022-outage-upda...

Hi, I'm Mike and I work in Engineering at Atlassian. Here's our approach to backup and data management: https://www.atlassian.com/trust/security/data-management - we certainly have the backups and have a restore process that we keep to. However, this incident stressed our ability to do this at scale, which has led to the very long times to restore.