What would you need it for? Once you defined your SLOs, either your service meets them or not. What's the value in alerting someone that "this graph looks funny"?
What would you need it for? Once you defined your SLOs, either your service meets them or not. What's the value in alerting someone that "this graph looks funny"?
Where automatic anomaly detection was very valuable for us was in the video domain with multi dimensional end user telemetry. I.e what would be lost as noise in top level metics could be surfaced via anomaly detection for specific combination of dimensions that you could not otherwise manually observe. I.e video start time in Mexico is fine ... But an ISP in Mexico City is not failing but when data is sliced and anomaly highlighted we see its newly under preforming and we need to feed this data into our CDN switching to improve video start time there.
The data had too many dimensions that were always changing with degraded experience easily lost in the noise when measuring across platform and our software updates, combinations target devices, connection types, geo location, specific content, active ab tests, etc. In such cases automatic anomaly detection was pretty critical.
What's the value in mentioning to someone that the chicken they're about to eat looks a bit raw?
It stops them from eating it and getting food poisoning.
Anomalies are often warnings (harbingers?) of a problem which could lead to a fault and downtime.
So doing it in a way where you can quickly scan such events i think could be useful even if only 1 in 20 actually turn out to be some potential problem that might have been missed by your alarms or maybe could even be a precursor to some impact on SLO's etc.
The aim would also be "this collection of graphs look funny at the same time" as opposed to "the individual graph looks funny" as if you have an anomaly score for every chart for sure at any given moment some individual charts would be randomly firing. But when you pool the information across charts and hosts and systems the hope is that then you can use anomaly detection as another way to explore your system and catch when things change unexpectedly.