Some thoughts on Prometheus Alertmanager's alert reminders

The root problem here is not a remind interval, but "email alerts".

Either something should file a bug to look at "some time later", or it should start ringing your phone.

Email alerts are just spam.

For both bugs and pages the reminder is good. A bug should get an update essentially saying "yes, this is still looking bad" (not create a new bug), and if it was important enough to page you the first time, then (unless you silenced the alert) it's important enough to remind you by ringing your phone again.

This is natively supported in PagerDuty, OpsGenie, etc. Routing, time-based suppression, alert reminders, etc. I’ve tried to implement all of that in Alertmanager itself, and it’s much easier to just swipe the card.

As for the dashboard, there’s an Alertmanager data source for Grafana that’s super easy to set up. If you have Grafana already I would just go ahead and do it since it takes maybe ten minutes.

I found myself nodding along with the author's musings and don't have much to add. I just want to say that I love their site. I'm putting together a personal site of my own and I want to steal some of their ideas, especially the "View Source" page tool at the bottom.

Perhaps severity should also influence how often an alert is repeated. And during a holiday break where no one is available to react to incidents, a alert silence should be active until people are back.