I have been on L2 and L3 on-call duty in my previous company, and some of the times these dashboards were life-savers.

Is the queue growing? Certain node not processing all it should? e.t.c

Alerts actually did go out of prometheus.

Hindsight is 20/20 and I remember i.e. having to change alerting on median aggregate node memory spiking to alert on any of the nodes spiking. Seems obvious in retrospect, but if we only have alert, I don't really know.

And there were parts where the dashboard wasn't obvious in retrospect, in the more complicated parts of our data-pipeline,ad there Graphana really shines.