8040

I would like to take this moment to really highlight "Recovery mechanisms should be fully tested before an emergency". As the unexpected SRE at Google who became known by entire company for using a double negative incorrectly, it is something very important to do right away.

For those Googlers curious, you can search my username internally for how I generated more impact then could be measured.

https://status.cloud.google.com/incident/google-stackdriver/... was one of the public incident reports around it.

while technically correct, it was degraded in a number of interesting ways that one might describe as broken, but still within SLO

I broke this once several years ago. I even use the incident number in my random usernames to see if a Googler recognizes it.