> On-call attempts to re-enable the R2 Gateway service using our internal admin tooling, however this tooling was unavailable because it relies on R2.
It's comforting to see this happen to a big tech co!
It's comforting to see this happen to a big tech co!
As far as I can tell from the timeline, it only took 11 minutes from the moment the on-call first attempted the action until the ops team began responding.
Given that this issue was caused by someone unintentionally using a level of access that they had to do something they did not intend, and the minimal impact reduction, deciding not to grant higher levels of access to the on-call seems to me to be the right decision.