You need 16 times the sample size to estimate an interaction than a main effect | Hacker News Reader

You need 16 times the sample size to estimate an interaction than a main effect

andrewgelman.com

142 pointsluu9 comments

That 16x number is specific to a set of assumptions in the article. Is this just the Curse of Dimensionality in action? https://en.wikipedia.org/wiki/Curse_of_dimensionality

Andrew calls it the blessing of dimensionality: http://andrewgelman.com/2004/10/27/the_blessing_of/

Multi-level modeling's concept of partial pooling is really a killer feature.

Bob Carpenter did a great case study on the this topic, for anyone new to it: http://mc-stan.org/users/documentation/case-studies/curse-di...

Check out the graphs, especially of the euclidean distance from the mode of a random point as dimensions increase.

It's part of it, but the described effect is amplified by the fact that if there are n variables, there are (n^2-n)/2 potential interactions. This increases the risk of finding spurious correlations due to multiple testing.

The most important part of this article is the way that Gelman derives the value. He writes a simple R script to simulate the outcomes. Many, many introductory stats classes will teach you a variety of formulas to estimate the "statistical power" of various experiments, and it's easy to get intimidated and worry if you know the right one to use. But if you know how to program, and you have a reasonable set of assumptions you'd like to test, then you can do a similar analysis in about 10 minutes. These analyses can often save you months of work when you figure out that your exciting study idea has no hope of detecting a difference or that you actually don't need 10000 subjects to detect a 10% lift in conversion rate. Thank you, Professor Gelman!

Further suppose that interactions of interest are half the size of main effects.

Why make this assumption? I'm hoping to see more justification as to why the main effect should be assumed to be exactly 2x the interaction effect.

Edit: In comments he wrote - I think it makes sense, where possible, to code variables in a regression so that the larger comparisons appear as main effects and the smaller comparisons appear as interactions.

This is an exam question, so the author is free to assume whatever he wants. But, more importantly, the interaction as large as half of the main effect is already pretty large -- in real world, many studies look at much smaller interactions. The point here is that if you need 16 times sample size to get enough power to detect the interaction at half effect size, you will need even larger sample size to detect smaller interactions.

If you're really interested, here are some other recent takes by statisticians on this issue: http://datacolada.org/17 https://approachingblog.wordpress.com/2018/01/24/powering-yo...

It's simply field experience that interactions are more typically subtler modifiers of strong main effects – here's a relevant quote from the second source:

>Unfortunately, very few times in psychology do we add a factor and expect a complete reversal of the effect. (...) [It’s] more usual that we expect the new condition to alter the size, rather than direction, of the existing effect.

Or from the first:

>[Whatever] power [simple effect study] had, at least twice as many subjects are needed in [interaction study], per cell, to maintain it. We know this because we are testing the reduction of that same effect.

Interactions are usually smaller than main effects, the idea that they are half the size is just a convenient size to get a point across. If you allowed the interaction size to vary between 1% to 99% of the main effect then you'd get such a huge range of resulting required sample sizes that it wouldn't be useful to illustrate his point- that interactions require a larger sample size to estimate than just tacking on another main effect. If you wanted to detect a main effect that is half the size of the others you only need (roughly) 4x the sample size, not 16x. Part of this is due to the way Gelman defines "interactions half the size of the main effects" but the idea that interactions are harder to estimate than main effects is true.

The assumption that interactions are smaller than main effects is generally true in real life. There's an idea in statistical modeling called the hierarchy of effects. Main effects are the largest while second-level effects are smaller and don't appear without the presence of the corresponding main effects. It's possible to construct datasets in which the hierarchy of effects isn't true (synergy only, without any benefit from a single variable), but it's uncommon in real life that you could have an interaction without both corresponding main effects being present.

Reply on news.ycombinator.com