this is detailed in the Google SRE book: https://sre.google/workbook/incident-response/