I use offset function in Prometheus to make an average of past weeks as a recording rule. We have a use in our systems that is very "seasonal" as in weekly cycles so I make an average of some metric (offset 1 week, 2 week, 3 week , 4 week/4) and I compare it to the current value of that metric. That way the alarms can be set day or night, weekday or weekend, and the thresholds are dynamic. It compares against an average of the day of the week, or time of the day. There is someone in Gitlab that posted a more in depth explanation of this way of working. https://about.gitlab.com/blog/2019/07/23/anomaly-detection-u... Things get a bit more complicated with holidays, but you can actually programm them into prometheus https://promcon.io/2019-munich/slides/improved-alerting-with...