Hey there! I work with alerting in general at Grafana - what are the pain points of dashboards and alerts as code you're currently experiencing? Would love to deliver / capitalise on the feedback.
Alerts are also integrated tightly in dashboards. Forces alerts to be saved/backedup/imported as single json blob. We want separate management of alerts so they can be defined as code and not in the dashboard blob of json!
What makes me chagrined is because of the above issues we have to use prometheus alert manager instead while our colleagues absolutely LOVE grafana itself! We can't duplicate alerts tens of tens times. We don't want that management nor do we want to teach our colleagues jsonnet/ksonnet to generate it. We also don't want permission problems.
Appreciate the second look and please let me know if you have any additional feedback.
I've literally copied an alert's json format, and then tried to post it back and never got it to work.
Here's an example from my bash history:
> curl -X POST -H "Authorization: Bearer $GRAFANA_API_KEY" -H "accept: application/json" -d @rule.json some_endpoint/api/ruler/grafana/api/v1/rules/test1
I spent a solid day trying to play around with this to get it to work. Because of this the alerts are impossible to code review or store in a git source. Which stinks because Grafana's datasource API's would be amazing to use for alerting. But they're either unusable because anybody can change them or the administrator could bork them at any given point (which has happened before), or just undocumented to the point where they are useless.
That's not even to begin on dealing with the "big blob of json" problem [1] that was clearly important enough to be given an entire spot at GrafanaCon, but even Grafonnet is not supported with Grafana 8. There is apparently some CUE way of doing this, but I can't seem to find any official documentation on that.
Anyways, I've moved back to alertmanager for the time being.
edit: is all of grafana labs downvoting the GP? this is very honest and candid feedback here.
[0]: https://editor.swagger.io/?url=https://raw.githubusercontent...
[1]: https://grafana.com/go/grafanaconline/2021/dashboards-as-cod...
Missing basic functionality like that is a dealbreaker.