Hacker News Reader

Comment by darkxanthos | Hacker News Reader

I do this professionally as my sole job. This is one of the very few papers I've read that seem completely legit to me. I especially love their point on necessary sample sizes to get to a 90% power.

View on HN

How do you calculate the correct sample size for a test, to achieve the correct "power"?

Here are a few resources: http://statpages.org/#Power

For binomial scenarios like a stock A/B test, most statistical environments will have some sort of built-in power functions. For example, R does; an example: http://www.gwern.net/AB%20testing#power-analysis

For simple tests you can reverse the mathematics to get good estimates of how many observations are needed given a goal for your desired power and tolerance for false positives. Asking for greater power makes your test more sensitive ("buys a bigger telescope") at the cost of increased sample size. Asking for fewer false positives ("cleaning the lenses") costs similarly.

For more sophisticated tests, ones less likely to be seen in an A/B scenario, you might not be able to reverse the mathematics and get a direct answer, so often people will run simulation studies to guess at the needed sample size.

Lehr's Equation (Rule of 16) is generally a good estimator. It is explained (but not referenced) in a similar article here: http://www.evanmiller.org/how-not-to-run-an-ab-test.html

I use simulations so I can avoid math and analyze the power of any distribution I can model in a consistent manner.

I wrote something about it here: http://www.databozo.com/2013/10/12/Finding_a_sample_size_for...

EDIT: typo

Why is 90% power the magic number? What's wrong with 89.99%? Or 99.99%?

It's not, you can use whatever power you want.

Nothing's "wrong" with 89.99% - 90% is simply a preferred value. The point of the exercise is not to pick a magic number, but to pick A number and therefore a target sample size before collecting results.

Without something like this, it's very easy to fall into the "we'll collect data until something that looks significant appears" trap.

Wait, nothing is wrong with collecting data until you see something significant. In fact, that's the better way to do things. You just need to make sure your math acknowledges the procedure.

In practice, doing this will lead to what the author talks about under "Stopping tests as soon as you see winning results will create false positives." It doesn't make your data invalid, but it will generally lead to poor methodology if you are not careful.

http://www.evanmiller.org/how-not-to-run-an-ab-test.html gives a good explanation about how this can ruin your tests.

He even says this is possible via Bayesian experiment design:

FTA: Bayesian experiment design: With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences.

> You just need to make sure your math acknowledges the procedure.

I'd love to hear some clarification on how "[making] sure your math acknowledges" stopping randomized tests early works, because it flies in a face of how actual statistics works.

> it flies in a face of how actual statistics works.

The terms you're looking for are 'sequential testing', 'optimal stopping', and 'adaptive trials'.

consider fair coin flip sequences. stopping at arbitrary points of your choice can never really affect anything serious. but you need to make sure you're doing math right, meaning if you stop when something gets a lead you don't use incorrect math that says the coin is biased.

data is data. as long as the method being used to collect individual data points is fine, and they are collected independently, then the data you get as a result is gonna be OK, the rest (like arbitrary stop time) doesn't ruin it. you just have to avoid bad math.

what ruins data is stuff like throwing 10% of the heads results in the trash or using other approaches in which data can be selectively discarded or not discarded. so just stopping arbitrarily can be a problem if you might never stop and throw out the results if you don't like them. but if you do something like "stop after 1 million data points max, or when i feel like it earlier" then your data is still OK because it cannot get selectively ignored.

stopping earlier cannot make a fair coin look unfair or anything like that.

this is not some random unknown position that flies in the face of how actual statistics works. something like this is the standard bayesian position, and i think it's true. (i strongly object to bayesian epistemology, but i think bayesian statistics is correct).

not ALL stopping rules are OK but lots are. you don't HAVE to use simple ones like "gather X data points, stop".

see stuff like:

http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-a...

http://lesswrong.com/lw/mt/beautiful_probability/

> And then there's the Bayesian reply: "Excuse you? The evidential impact of a fixed experimental method, producing the same data, depends on the researcher's private thoughts? And you have the nerve to accuse us of being 'too subjective'?"

Previous HN discussion: https://news.ycombinator.com/item?id=3188816

By "acknowledge" you probably mean apply an SPRT (serial probability ratio test). The Neymann-Pearson lemma is also relevant.

Setting a 90% power target is a way to decide when you have something significant.

Tests are calibrated on their false positive (alpha) and false negative rates (beta). If you have a lot of financial/upside/pain information then you can start to determine the relative pain of each of those kinds of failures and calibrate accordingly. At the end of the day the best choice is some complex function of the cost of false positives, the cost of false negatives, the cost of each new observation (which is probably non-linear), the upside of a discovery, and the prior likelihood of finding a discovery.

Which is to say you can definitely pick better choices for alpha, but it's really hard so everyone just picks whatever their field agrees is "OK". In science it's often 95%.

Reply on news.ycombinator.com