Hacker News Reader

Comment by baq | Hacker News Reader

IMHO plotting the distribution should be the first step before trying to compute its statistics. If you know the shape, you can understand the values - otherwise it's guesswork.

View on HN

Agreed, this is actually demonstrated by Anscombe's quartet, a set of "four datasets that have nearly identical simple descriptive statistics, yet appear very different when graphed" (https://en.wikipedia.org/wiki/Anscombe%27s_quartet)

... although median and percentiles would be different within the Anscombe's quartet.

But there are other datasets that would do the same thing with medians.

We've switched to box and whisker plots for most of our statistical reports, they give you a good idea for various important percentiles and adding average indicators is fairly simple if desired. Even for things like latency this can be quite useful.

Box plots can still obscure the nature of a distribution, e.g. it might be multi-modal.

Violin plots + outlier dots + additional markers are more helpful. Sometimes the CDF is also more useful than the PDF, e.g. for latencies.

I wholeheartedly agree! Violin plots are a great way to get a dense appreciation for a distribution. Multi-modal distributions are completely masked by mean, even median+std.

This entire thread is great, 10 data scientists all want their own special chart to be the best. You are all wrong, you should have a view of all these charts!!!! hahaha

Seems like a good idea, I wish I had more of a statistical background for this kind of thing. Seems like it'd have proved more useful for most software purposes than the calculus I had to take instead as a prerequisite.

It's basically just my boss making suggestions and me implementing, so the results are probably less than optimal for this kind of thing.

Even more generally speaking, there's more than a few people saying we should elevate learning statistics over calculus in the general maths curriculum. Arguably it's even more important than trig and geometry.

[0] is just one of the first results from a search that makes this case.

[0] https://stanfordreview.org/calculus-is-overrated-why-we-shou...

I dislike violin plots because they don’t give a good sense of how many points there are overall, and this can be very misleading if you’re trying to compare segments of different size. They also look like female genitalia and I’m 100% serious when I say this tends to distract people for laughs.

I like swarm plots for this kind of task.

https://seaborn.pydata.org/generated/seaborn.swarmplot.html

Edit: well, I suppose I should clarify the comment on violin plots is implementation dependent and biased by my personal preferences for visualization libraries

The plots on that page seem to exhibit some clustering and weird shapes which might not be all that great for discrete data or if you have sampling artifacts since it distracts from the overall shape of the distribution.

Showing the number of samples is good though. The combined version seems useful.

> don’t give a good sense of how many points there are overall

Consider inner="stick" from [1] instead. It probably comes closer to what you're looking for.

[1] https://seaborn.pydata.org/generated/seaborn.violinplot.html

Violin plots are harder to parse at a glance.

Like all data visualization techniques, which plot type is better depends on the context.

I find raincloud plots to be the best of both worlds, and they have been my preference lately:

https://wellcomeopenresearch.org/articles/4-63/v1

They can visualize common statistics like box-plots, the distribution shape like violin plots, as well as the raw data!

Oh, that is quite nice. Communicates a lot in a single chart.

Or a similar is to violin plots - heat map / waterfall.

Not a bad idea, but in my line of work, tools like statsite calculate the summary stats and throw the data away because aint nobody got storage for that. From this viewpoint, it's still important to teach engineers the mean is heavily influenced by outliers. If they're looking to know what the 'average' is, a median is often better, and if they want to know how broad the distribution is, p95/p99 is a good slice into the long tail.

I'm not sure that's really relevant here, since this is monitoring over time. When you design this system sure you could look at the graphs of data, and you'd still end up with the same impression. The point is that over time you can't collect and analyse all the data always.

The important thing here is understand the underlying system and identify what it's modes of failure are. If the likely mode of failure was just to fail to serve some requests at all then the new approach in this article would be entirely useless since the latencies would be the wrong metric completely.

In this case they identified one mode of failure is for a small % of requests to deteriorate almost exponentially and so watching the 99th percentile is useful - this is very common in networking. That's great for this system but the overall message is to find the metric that actually reflects the ways your system will fail- and you need to do that in a practical way that may not necessarily involve recording and storing all the data.

> IMHO plotting the distribution should be the first step before trying to compute its statistics. If you know the shape, you can understand the values - otherwise it's guesswork.

My manager's dictum is histogram and time series to start, which falls under the same auspices.

Kibana can do visualizations of distributions with elastic data. I agree it is the best way to get a clear picture of latency and other metrics within a period of time.

I think percentiles over time, given consistent sample sizes per bucket, can be good for understanding change over time.

In general I think of histograms for these sort of metrics providing higher fidelity of value, where as stats over time can provide higher fidelity of time (for roughly the same storage).

Animating the distribution over time can be pretty neat, but not common feature in my space (but can be done with a little R code).

I imagine 3 axis (3d) histograms over time can make some neat visualizations but have never experimented with that, and would probably want to slice or rotate the plot. More common are heatmaps where the darkness of the color is your third axis. Kind of like viewing the 3d plot of histograms over time from above where the cubes are partly transparent and get darker the deeper they are.

Reply on news.ycombinator.com