We generally targeted specific statistics of derived/processed streams. For some such streams we cared if the mean changed. In others if the spread changed in a way that was unusual for the time of day. In yet others if some percentile changed that was unusual for the time of day. Sometimes it will be more than one of such statistics.

Then we would track an online estimator of that measure with an SPC chart. The thresholds would be set based on our appetite for false alarms. We did not fit or use properties of parametric distributions that standard SPC charts use. So no 3-sigma business. In our case convergence to Gaussian would often be not fast enough for such techniques to be useful.

Also the original streams were far from IID, temporal dependencies were strong. So we had to derive from them derived streams that didn't show temporal dependencies any longer, at least not as strongly. This was the most important bit.

The next key aspect was to keep the alerting thresholds as untarnished and unaffected as possible from the outliers that would unavoidably occur. Getting this to work without additional human supervisory labels was the next most important part.

Make this part too robust to outliers then the system would not automatically adapt to a new normal. Make it too sensitive and we would get overwhelmed by false positives.