Hacker News Reader

Comment by i_love_limes | Hacker News Reader

So, it is like that, but it's also not. A couple decades ago, with the advent of GWASes, because of all the multiple testing that was going on, there was an agreed new p-value threshold p < 5e-8. This was to account for multiple testing going on (how that number came to be requires more explanation of LD + other things).

That is the minimum threshold. This study found that peak was at p < 1e-37 or so. But that is where the biological analysis begins. Unlike social scientists, we don't stop with the statistical correlation, we then go on to look at what we know about that gene, the type of mutation, if it's a loss or gain of function, what role that gene has in various tissues, etc. And mendelian randomization is another way to unpick the causal direction of effect.

Not to say this is the truth or causal, but it's a lot closer to causal than what you are implying.

View on HN

In other words, there is a likelihood of 1/10,000,000,000,000,000,000,000,000,000,000,000,000 that the statistical result was due to random chance.

In more casual sciences p < 0.05 is considered the limit of significance, i.e. less than 1/20 likelihood of statistical testing favoring the tested hypothesis over the null hypothesis due to random chance

If you are testing a single hypothesis, sure. But nowadays, statistics training really weighs on bonferroni correction, or other methods to deal with the issue raised by the above-referenced XKCD, whentesting multiple hypotheses.

> This study found that peak was at p < 1e-37 or so.

If true, this would be cause for someone to read through the study to check there are no maths errors, and if it holds up then to take action immediately.

This isn't a "wait for more science to come in and confirm" type thing.

> This study found that peak was at p < 1e-37

This is many orders of magnitudes better confidence than any physics experiment, it feels unlikely a biological result can even be this strong, so it makes it sound like a statistical error.

Bioinformatician here. These kinds of p-values are common in these kinds of experiments (GWAS, or association studies), and happens almost automatically once you get enough statistical power. The big problem is that once you have so much statistical power, you get very small p-values from small effects, and then the often-overlooked assumptions behind frequentist statistics begins to matter. Are your samples _really_ idenpendent and identically distributed values? No. Are they really normal distributed? No.

Also, things like gut microbiome and depression are linked through what some people call the _crud factor_, which are weak correlations between nearly all social aspects. For example, probably depressed people eat differently from non-depressed people, causing changes in their gut microbiome. Probably, there are variations in human population's depression rates and obesity rates (correlated with gut microbiome) that somewhat correlate. When you have enough statistical power, you see the crud factor everywhere.

I really appreciate your explanation regarding the crud factor. It adds a lot of intuition.

Out of curiosity, what is your perspective on gnotobiotic systems? I distinctly recall an example of a gnotobiotic mouse that, upon being provided a microbial sample from an obese mouse, started to have substantial increases in body fat despite a simultaneous reduction in feed. Would that type of experimental approach still run into the statistical difficulties you mentioned?

Well no. It's not uncommon for p-values to be even lower than that. We are talking about a specific SNP (allele) having a specific mutation being predictive of a phenotype / outcome.

So, a specific SNP mutation being predictive of a gene expression / protein is basically a p-value of 0.

Can't speak for physics experiments, but this is almost certainly not a statistical error

Reply on news.ycombinator.com