Hacker News Reader

Comment by Xcelerate | Hacker News Reader

You know what I've always found really useful? Entire distributions. Forget means, medians, percentiles, etc. — give me the complete distribution (along with sample size) so I can understand all of the nuances of the data.

(Better yet, just give me the raw data so I can analyze it myself. I find it hard to blindly trust someone else's conclusions considering all of the p-hacking going on nowadays.)

View on HN

Yeah. We're starting to contemplate a new monitoring/performance/etc. system, and I'm thinking I don't want to do any of the long-term storage aggregration like in rrd. We'll store the full data for each item forever. Not sure if we can afford the storage, but I want to try for a while and see if it works.

We did something similar at Netflix. We had all the aggregations but also stored all the raw data. The raw data would be pushed out of the live system and replaced with aggregates and then stored as flat text in S3. If for some reasons you needed the old data, you just put in a ticket with the monitoring team to load the data back into the live system (I think this is even self service now).

The system would then load the data from S3 back into the live system via Hadoop. Turns out it was pretty cheap to store highly compressible files in S3.

I've built a similar system as well. Raw data compressed and stored in s3, but aggregations are stored in postgres. Data stored in Postgres is compressed binary representation of a histogram and I added few C functions in postgres to do things like select x,y, histogram_percentile(histogram_merge(data), 99) group by x,y etc..

My company stores all raw data for pretty much everything forever (and, in finance, there are a lot of things). It's binary, compressed with xz, and stored in several places. The grand total Glacier cost is something like $50/month.

(Do NOT, for the love of your sanity, emulate this design exactly. Use Backblaze or Google's nearline storage. Do not touch Glacier at all if you can avoid it. When I wrote the Glacier integration, I did it because it was far cheaper than its competitors. That's no longer the case.)

Funnily enough, I am writing an article about this now.

FYI I don't think its right to say "forget means, medians, percentiles, etc." They are hugely useful.

You just have to note that they aren't everything. As you've noted, only the raw data is everything.

But the raw data isn't everything either, it's usually just a sample. The analysis of it is there to explain what the data means.

If you're saying that people should provide the full data along with their analysis, I fully agree! But if you're saying "skip the analysis", then unfortunately I, and most people, are not statistically educated enough to do all the analysis ourselves.

Some caution is warranted.

Always keep in mind that estimating distributions from samples is very hard, in particular for multidimensional and continuous variables. Sometimes even infinite samples do not suffice (I am not kidding).

The good news is that there are interesting and relevant tasks (for example classification) where you do not need to know the distribution to solve the task optimally, although if you somehow managed to do so (find the distribution) you would be able to solve the task optimally. Or more succinctly, knowing the distribution is sufficient but not necessary for such tasks (small mercies)

It's also often useful to think of the true distribution of the underlying system/dynamic in question, and to consider a histogram as a sample that therefore has sample error. E.g. if the histogram's tail bins have very few samples then they're giving you some information about the shape of the distribution in that region, but not the exact probability of events in that region.

This is an excellent point. The distribution of the measurements are the distribution of the sample, not of the population. In my experience very few non-statisticians understand the difference. Statistics is hard.

This is true. I almost mentioned the difficulties of estimating the kernel density bandwidth in my original post, but thought that would be getting too far on a tangent.

Your post is a little confused. For actual data, there's no distinction between the empirical distribution and the data, so when you want "along with sample size" with your "distribution", that's what you're getting. (same with your "raw data").

A nice distribution diagram is very informative but if you want to track the performance through time and have warnings triggers in place you need to be able to capture what you care about in a handful of numeric metrics.

Reply on news.ycombinator.com