A great doc from Google that they didn’t link to above is “My Philosophy on Alerting”, by Rob Ewaschuk (https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa...). A must read for anyone setting up alerts IMO.
A great doc from Google that they didn’t link to above is “My Philosophy on Alerting”, by Rob Ewaschuk (https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa...). A must read for anyone setting up alerts IMO.
I've been with teams who treat alerts as informational and then quickly get alert fatigue. Then when something actually does happen they don't recognize it because they've trained themselves to ignore all alerts.
At my current company we regularly review our alerts to make sure they are actionable and meaningful still.
“Something went wrong, but at acceptable levels/without much user impact” belongs in dashboards, not alerts. Worth being able to monitor, but alerts are for urgency, and that’s not urgent.