Hacker News Reader

Comment by sethammons | Hacker News Reader

I would argue they should be considering max. Percentiles are good for understanding macro trends. But if every request is sacred, max needs to be monitored. Even at four nines, there are a couple dozen requests per second that are greater than what you are observing. How much greater? Who knows, if you are not observing max. Could be minutes in latency.

View on HN

Max is pretty horrible, as any comparison requires identical cohort sizes. P99.99 is much better if you want to go wild.

Given that the service probably did not have a 100% success rate, and almost certainly had timeouts, the "max" would also likely be at the timeout.

And that would be a good signal and worth investigating. Why timeouts? I run a few services doing billions of daily queries. I watch max in addition to percentiles.

> and almost certainly had timeouts, the "max" would also likely be at the timeout

You filter out the 504s, just the same way you would analyze it on a per-route basis for those metrics.

Sure but now your "100th percentile" would be nothing but the timeout.

Quantifying things like service latency isn't a one-size fits all thing. Every service has its nuances and use cases that make it more meaningful to measure 99%, 99.9%, 99.99% or something else.

.... just don't measure average like I've seen naive junior devs do. Average latency is the worst of all metrics to use as it will include all the outliers at the very tippy top of the spectrum and basically render the metric meaningless.

Reply on news.ycombinator.com