Hacker News Reader

Comment by nickledave | Hacker News Reader

See mattkrause's comment below. I think you might not be understanding what they're testing. They're taking only resting state data, randomly sorting some of it into "pretend active state data" and then asking whether they get any statistically significant difference between these two groups when they shouldn't. But they do. That means the tests they used, the same tests many authors use, are giving false positives. The null hypothesis is "there will be no difference between testing state data and whatever data we get when we ask the subject to do some activity". They can "reject" that hypothesis using only randomly shuffled resting state data, so there's something wrong with the stats packages themselves

View on HN

I think that like mattkrause (and the authors of the current work), you have forgotten that the null hypothesis is something larger than condition 1 == condition 2. There are various other components, usually (somewhat misleadingly) referred to as assumptions, that can also cause the predictions derived from the null model to deviate from the data.

For stuff like a t-test, one parameter value (ie the mean of the distribution) gets all the attention. But this is wrong, it is only one part of the model being tested.

I'm aware that there are assumptions implicit in what the null hypothesis is. You are the one who keeps saying the authors don't even realize what those assumptions are, but you haven't pointed out anything besides what the authors said. What are the other faulty assumptions you've identified that the authors are missing? I guess you're referring to some sort of issue with power that you mentioned in your previous comment?

>"You are the one who keeps saying the authors don't even realize what those assumptions are, but you haven't pointed out anything besides what the authors said."

I am saying they are confused because they say "the null hypothesis should be true" under their conditions, when they know for certain that it is false! Therefore these "false positives" are not false at all. They are totally legit "true positives".

These authors are blaming the statistical test when the problem lies with their crappy choice of null hypothesis.

There may very well be other issues, but I have not inspected the code or done anything other than read the description in the paper.

Okay, why do you think the null is crappy?

In the absence of any information whatsoever, the idea that an analysis pipeline produces false positives at or below its nominal rate seems pretty reasonable.

But, they have some prior information.

Let's look at the E1 paradigm (2 sec on, 6 sec off). In the NeuroImage Paper (Figure 1A, 2A), the FWER on voxel tests is statistically indistinguishable from 5%. In other words, it's appropriately sized. They replicate this result in the rightmost panel of the PNAS paper, where it's also within the 95% CI around 5%.

Now, for the cluster inference, look at the left-most panel of PNAS Figure 1A. The E1 paradigm is the green bar. Using the defaults for FSL (left panel) and SPM (middle panel), the FWER is about 30% and 25%, respectively. That is not good.

I agree that the Block designs look awful in the NeuroImage paper, which makes it hard to say whether this phenomena makes it worse in the PNAS data. It's unfortunate that these numbers are going to be in the press release (70% is much sexier than 30%), but 6x the nominal rate is still bad.

>"Okay, why do you think the null is crappy?"

Because their goal is to determine if some sort of treatment has an effect. If the null is false for other reasons, then statistical significance can't be used to support the existence of a treatment effect. So these would be pointless, pedantic calculations.

@nonbel: Are you saying that an fMRI, when taken of a subject at rest, twice - then what the data here show - is that this is likely to be interpreted as a subject in two different states? And the researchers are ignoring the fact, rather insisting that we should be able to tell that these two states are the same (and then, perhaps tell them from other states)?

I see a few ways this could come about: perhaps the way we record and model activity doesn't conform to the distribution we assume (I'm not sure if they assume a normal distribution here - or if that even makes sense given the nature of the data) -- or perhaps the issue is with taking 3d/4d data and "turning it into" an easy-to-model statistical model (like the normal distribution)?

At any rate, it does seem that they're saying we can't tell that one individual at rest, measured twice, is in the same (rest) state both times? Hence, they're null hypothesis is bunk?

>"Are you saying that an fMRI, when taken of a subject at rest, twice - then what the data here show - is that this is likely to be interpreted as a subject in two different states?"

Yes, that is what they seem to be saying. I didn't read the code, or even the paper very closely. However, from what I quoted, they seem to be saying there is some assumption about autocorrelation that introduces what they call "false positives".

I am saying they have mischaracterized the problem. These are true positives.

@mattkrause: you beat me to it

Eh, you'll scoop me on something when it matters.

PS: If you are the song bird guy, I saw a talk of yours on youtube. Cool stuff.

Reply on news.ycombinator.com