Hacker News Reader

Comment by yummyfajitas | Hacker News Reader

Most classical tests don't require the variable to be normally distributed, they require the test statistic to be. I.e., you don't need revenue/session to be normal, you need sum(revenue_per_session) to be normal. As long as you don't have any long tails and your variables are IID, that will happen: https://en.wikipedia.org/wiki/Central_limit_theorem

More interestingly, things like revenue/visitor have a known probability distribution. It's not normal, but it is known. You can use a LOT fewer samples if you use a parametric test (either Bayesian or SPRT) based on the correct distribution.

If you use bootstrapping instead, you'll a) give up all your finite-sample guarantees and b) wind up using a LOT more samples than you need.

View on HN

But how useful is comparing sum(revenue_per_session) when you want to test significance of one batch to the other? Aren't you then just comparing 2 values and seeing which is greater?

If you compare the 2 batches of revenue/session distributions using a monte-carlo simulation you can calculate the probability that one is significantly different than the other. This generalizes beyond the 2 sample t-test because those underlying distributions are non-normal.

Please let me know if I'm thinking of this correctly (or not)

Ok, to test one relative to the other, you might test W=sum(revenue_per_session_A - revenue_per_session_B). Interpret the subtraction as a vector op. (Adjust a bit if you want to do a Welch test.) Assuming the CLT holds this statistic is normally distributed. Assuming the null hypothesis holds, it has mean 0.

Thus, you can do all your normal Stats 101 tests on it.

If you compare the 2 batches of revenue/session distributions using a monte-carlo simulation you can calculate the probability that one is significantly different than the other.

A frequentist test (which includes most bootstrap methods) can never tell you this. Frequentist statistics doesn't even acknowledge this as a legitimate question to ask.

Now I agree, if you can use the exact distribution of revenues directly in the test, you can get answers even before you have enough samples for the CLT to apply. But if you use a nonparametric method like bootstrap, you'll need to use up a lot of samples unnecessarily.

>More interestingly, things like revenue/visitor have a known probability distribution.

This actually depends. Based on my experience at a very early stage startup, it was definitely not the case for some datasets (I even tried fiddling with various well-known distribution's parameters).

If I recall correctly, I believe that the bootstrap had some asymptotic guarantee on the rate of convergence (although my memory is hazy on this)?

EDIT: never mind, it is asymptotic, hence not finite-sample necessarily.

Reply on news.ycombinator.com