Somewhat famous p hacking example iirc
The authors are pretty clear that they didn’t find what they were looking for and so they just tried whatever they could with the remaining data and landed on dog pooping orientation.
When I put it like that, it evokes more pity than outrage. Imagine what they had to go through to collect enough data.
Last time I looked into it, I couldn't understand why it was seen as a problem.
The problem is if what you actually did wasn't one experiment, but 1000 simultaneous experiments. That happens quickly if you start looking for correlations in your data. The issue is that if you then use the same 1% threshold for significance, on average 10 experiments will clear that threshold by pure chance. Your statistical test has become worthless. That's fine if you account for it, but if you don't and just publish those 10 results that's called p-hacking. People hate it because it makes purely random results look as if they were significant.
You do your experiment. You don’t find any evidence you were correct. So you test 20 other ideas. Lo and behold, 1 out of the 20 new unregistered ideas passes the test of seeming to occur with less than a 1 in 20 probability of occurring.
Well duh. It would be weird if you didn’t find such an outcome. This particular study openly acknowledged doing this, and also did some creative binning of their numbers to help nudge the stats.
You will always be able to find something. P hacking doesn’t mean the thing you found is false, but it is at best a sloppy estimate of something to explore further.
The more egregious examples of p hacking is adjusting outliers and filtering out specific datapoints until you can make the data claim whatever you want.
> You do your experiment. You don’t find any evidence you were correct. So you test 20 other ideas. Lo and behold, 1 out of the 20 new unregistered ideas passes the test of seeming to occur with less than a 1 in 20 probability of occurring.
I think this ^^^ is the part that's tripping me up: the notion of "registered ideas".
Suppose we have 21 researchers, each testing one of those 21 ideas. They pool their funds and hire someone to generate that one data set.
20 of those researchers would find their ideas unsupported by the data. But one researcher would find their idea was supported. And that's the only study that gets published in Nature.
I don't hear about this ^^^^ issue being criticized the same way P-hacking is. But AFAICT it's a similar kind of statistical factor.
I'd think that (the extent to which data support a particular model) is unrelated to (subjective details such as what hypothesis the researchers intended to test).
IIUC, the criticism of P-hacking hinges on such subjective details.
If the main goal is to find useful models, why would we care about the researchers' intent when originally designing the experiment?
Further a lot of science finds things pass the statistical test but are still not useful, which is a different kind of problem.
If you do an experiment and see something interesting you should probably follow up on it. But it’s the difference between saying “there’s a 5% chance that this is just sample bias” using basic methods and “there’s a 90% chance this is sample bias using methods that account for how many things we tried”. Who knows? Maybe your finding is novel, important, statistically strong, and useful, without you having planned to explore it. Maybe you accidentally found penicillin.
The two big problems is that people don’t know how to do these calcs (a lot of which don’t really make any sense because they try to assert independence into things where independence is impossible) and they don’t mention the things they tried which didn’t work which makes evaluating their claims impossible.
So is the fundamental issue that an experiment is probably designed to avoid bias for the data related to the hypothesis being tested, but probably isn't designed to avoid bias for other hypotheses you might generate once the results are in?
Think of sample bias as a dice roll. Every time you check a new hypothesis, you roll the die. If it rolls a 1, you get a false positive.
Pre registering a hundred different things you want to test doesn’t make that better, but it does create confidence that you didn’t continually test new ideas one at a time until number 100 was something that looked good.
(I have this in my head when p-hacking comes up)
The real problem is when you set out to test 20 things, and they don’t work, so you test 20 more things; and then much worse when you only tell us about the analysis that lead to the 1/40 that mattered.
The thing about p hacking specifically is changing your analysis until it finds something that looks good. A lot of people don’t understand math and will do this and not understand that they’re gaming it. A lot of people will understand this and do it anyway because academia is soul crushing and they need to publish.
Sure.
When your experiment has a P value of 0.05, what that means is that 5% of the time you expect completely incorrect results purely due to random chance.
So you test $THEORY, and expect that 1 out of every 20 times the results are going to be wrong purely due to random chance.
P-hacking is when you repeat the test 20 times[1], and publish that 1 random result while throwing away the other 19 results.
[1] Or however many times you need to