Hacker News Reader

Comment by thinkmoore | Hacker News Reader

You might be interested in the statistical technique called "bootstrapping": http://en.wikipedia.org/wiki/Bootstrapping_(statistics)

The "A/A" method described is not a terribly robust way to estimate variance, but the basic idea of using subsamples to estimate variance is what bootstrapping does more systematically.

View on HN

I agree. Bootstrapping is somehow not covered in most "first course in statistics" yet it is very valuable in practice for data people, especially at startups: Because you don't have that much data, and most of your data do not naturally follow anything like the normal distribution, it could be misleading to use the normal theory to estimate variance. The bootstrap helps you combat this.

Yes bootstrapping should be introduced far sooner. Most of the variables we are interested in, say revenue per session, are in no way normally distributed, thus violating the assumptions of classical two-sampled t-tests. Bootstrapping, and MC methods, provide a better solution than parametric tests.

Most classical tests don't require the variable to be normally distributed, they require the test statistic to be. I.e., you don't need revenue/session to be normal, you need sum(revenue_per_session) to be normal. As long as you don't have any long tails and your variables are IID, that will happen: https://en.wikipedia.org/wiki/Central_limit_theorem

More interestingly, things like revenue/visitor have a known probability distribution. It's not normal, but it is known. You can use a LOT fewer samples if you use a parametric test (either Bayesian or SPRT) based on the correct distribution.

If you use bootstrapping instead, you'll a) give up all your finite-sample guarantees and b) wind up using a LOT more samples than you need.

>More interestingly, things like revenue/visitor have a known probability distribution.

This actually depends. Based on my experience at a very early stage startup, it was definitely not the case for some datasets (I even tried fiddling with various well-known distribution's parameters).

If I recall correctly, I believe that the bootstrap had some asymptotic guarantee on the rate of convergence (although my memory is hazy on this)?

EDIT: never mind, it is asymptotic, hence not finite-sample necessarily.

But how useful is comparing sum(revenue_per_session) when you want to test significance of one batch to the other? Aren't you then just comparing 2 values and seeing which is greater?

If you compare the 2 batches of revenue/session distributions using a monte-carlo simulation you can calculate the probability that one is significantly different than the other. This generalizes beyond the 2 sample t-test because those underlying distributions are non-normal.

Please let me know if I'm thinking of this correctly (or not)

Ok, to test one relative to the other, you might test W=sum(revenue_per_session_A - revenue_per_session_B). Interpret the subtraction as a vector op. (Adjust a bit if you want to do a Welch test.) Assuming the CLT holds this statistic is normally distributed. Assuming the null hypothesis holds, it has mean 0.

Thus, you can do all your normal Stats 101 tests on it.

If you compare the 2 batches of revenue/session distributions using a monte-carlo simulation you can calculate the probability that one is significantly different than the other.

A frequentist test (which includes most bootstrap methods) can never tell you this. Frequentist statistics doesn't even acknowledge this as a legitimate question to ask.

Now I agree, if you can use the exact distribution of revenues directly in the test, you can get answers even before you have enough samples for the CLT to apply. But if you use a nonparametric method like bootstrap, you'll need to use up a lot of samples unnecessarily.

I had the same thought. The bootstrap is a really simple and easy-to-implement technique that we've had for decades.

There is even some recent work on a "Big Data" (distributed) version of the bootstrap from Michael Jordan's group [1]. It's also pretty easy to implement and can be really useful in practice.

[1] http://onlinelibrary.wiley.com/doi/10.1111/rssb.12050/full

I use bootstrapping all the time. Such an easy way to estimate variance of your mean, median, variance, etc. The most important issue I've come across is to make sure that your experiments are really as random as you can get. Otherwise you can end up with biases due to systematically picking outliers.

Reply on news.ycombinator.com