Comment by btilly | Hacker News Reader

Comment by btilly | Hacker News Reader

It was an interesting read, but the author showed absolutely no clue why pre-registration was important. The point is to avoid selection bias. If the same exact study is done 100 times, but only the best 50 results get included in your study, a meta analysis of those results is going to show a strong statistical bias. No amount of waving around the word "Bayesian" can fix this - the only way to fix it is pre-registration of honest results.

Why am I so certain of this? Because of the following passage:

By my count, Bem follows all of the commandments except [6] and [10]. He apologizes for not using pre-registration, but says it’s okay because the studies were exact replications of a previous study that makes it impossible for an unsavory researcher to change the parameters halfway through and does pretty much the same thing.

But that misses the whole point. Whether the studies had the same or different parameters is irrelevant - the fact that you see the ones that worked out and don't see the ones that didn't is the only thing that matters. And the fact that the meta analysis didn't get this right makes the output statistical garbage. No further discussion needed.

To give a concrete example, let's say I want to show that I can influence a coin flip with my mind in order to produce a heads result every time. I do this by publishing trials where each instance involves me flipping a coin four times.

I conduct 16,000 trials. On average, 1,000 of them will show that I can flip a coin and get heads every time. If all I publish are these trials, that yields the amazing result that I can, in fact, influence the coin! Furthermore, the p-value is phenomenally small. And it's very easy to design this experiment to meet all the criteria mentioned in the post other than preregistration.

> It was an interesting read, but the author showed absolutely no clue why pre-registration was important. The point is to avoid selection bias.

You've missed his fundamental point: the mechanisms he listed like pre-registration are excellent for defending against plucking signal out of random sampling error. They do nothing for when you are investigating a real effect which is not what you think you are investigating: the point of the spooky studies listed is that in the psi experiments, there is still a signal of something. We don't think it's actual psi in the sense of supernatural activity, but the alternate explanations seem almost as bizarre and have almost as painful implications.

I don't think the btilly thought he was refuting the article, just commenting on a not-very-central-but-important misperception that it seemed to contain, and it seems to be accurate - implicit precommital to experimental details in a replication solves some of what pre-registration is about but not all of it and arguably not the more important bit. This seems to be true whether or not the article's thesis holds, and this being true doesn't undermine the article's thesis.

I did not miss his fundamental point. I was making a different fundamental point about why I view virtually every meta-analysis out there to be even more suspect than regular research.

It was the point which is most commonly pointed out by Randi et al, but which would be most worrisome for the author because it throws into serious question the kind of research that he thinks should be most reliable.

> I did not miss his fundamental point. I was making a different fundamental point about why I view virtually every meta-analysis out there to be even more suspect than regular research.

It certainly sounded like you missed it, but regardless: meta-analysis is well-aware of the selection issue. Dealing with that is half the point of meta-analytic techniques - p-curves, heterogeneity, funnel plots, the binomial test, trim-and-fill, etc. Publication bias is an issue which has been repeatedly quantified, and these techniques work reasonably well: when meta-analyses are compared to very large RCTs where selection issues are not a concern, the agreement is pretty good (confidence intervals are blown a bit more than they should be, but not horribly so). So you're either ignoring OP's very interesting points or you're tendentiously overrating an issue. Neither is a worthwhile comment.

There is this rather amusing section, though:

"(Richard Wiseman – the same guy who provided the negative aura for the Wiseman and Schiltz experiment – has started a pre-register site for Bem replications. He says he has received five of them. This is very promising. There is also a separate pre-register for parapsychology trials in general. I am both extremely pleased at this victory for good science, and ashamed that my own field is apparently behind parapsychology in the “scientific rigor” department)"

After reading that article, I actually came away with a very positive view of the field of parapsychology in general. Of course psychics aren't real, but the Flying Spaghetti Monster can be used to make legitimate points about the study of religion, to use an analogy.

He addressed ways other than pre-registration you can deal with selection bias:

7. Address publication bias by searching for unpublished trials, displaying funnel plots, and using statistics like “fail-safe N” to investigate the possibility of suppressed research.

It is, of course, possible with cleverness and organization to get around that sort of analysis in a way that it isn't with pre-registration, but that requires an actual conspiracy rather than just not publishing negative results.

...but that requires an actual conspiracy...

No, it doesn't require actual conspiracy. All that it requires is that a subpopulation of researchers has consistent biases which you didn't completely correct for. Given how statistics works, the larger the meta analysis, the smaller the portion of biased researchers you need to come up with compelling statistics.

Reply on news.ycombinator.com