The Google SRE book (online here: https://landing.google.com/sre/sre-book/toc/index.html) might be useful. Specifically chapters 6 and 10 on monitoring and alerting.