> In practice, teams continue to ignore this problem and instead rely on aggregate time-series that are essentially impossible to interpret correctly, such as “maximum of 99th percentile latency by server.”
I get that we can't know P99 across all servers. Let's say we're talking about latency and the max of P99's will be an upper bound on the P99 across all servers. Why isn't this true or useful?
The problem is that what people are usually asking for, is the 99th percentile of response time for user requests, and they get the above statistic, not because they want it, but because they are limited by their monitoring systems; they accept it as a proxy for what they want, when it is actually something completely different, and may have completely different values.
It would be more useful to calculate the distribution for overall response time, and also calculate the distribution for each individual machine; you could then look for anomalies (in your statistics) that could identify which machines are performing badly.
While it doesn't solve the alerts, for viewing the data like that heatmaps are amazing. But you need the raw numbers (or predefined bands for aggregation, but that limits flexibility)
If it were me, I'd probably start by caching the slow requests (that are above the p99) first & adjusting the alert thresholds to match the new performance.