Hacker News Reader

Isn't that just a bad study? You have confounding factors - such as ethnicity - that weren't controlled/considered/eliminated.

I do get what you're saying, if you miss something in the study that is important, but I don't see how this is a case to drop the value of statistical significance?

In medicine, it is essentially impossible to control for all possible factors. Case in point, ethnicity is not biologically realized either; it's a social tool we use to characterize broad swathes of phenotypic [and sociocultural] differences that are more likely (but not guaranteed) to occur in certain populations. But the example provided of indigenous arctic people is itself imprecise. You can't control for that, not without genetic testing - and even then, that presupposes we've characterized the confounding factors genetically, and that the confounding factors are indeed genetic in orgin at all.

Put another way, the population is simply too variable to attempt to eliminate all confounding factors. We can, at best, eliminate some of the ones we know about, and acknowledge the ones we can't.

> ethnicity is not biologically realized either

What does this mean? Is it contrary to what OP is saying above?

Not exactly. What I mean to say is this: We know there are certain phenotypes that predominantly appear in certain populations, in broad strokes. But while we do have lists of correlates, we don't have good definitions for what an "ethnicity" is biologically, and there is very good reason to believe no satisfactory definition exists.

To use OP's example, we know that the gene mentioned is frequently found in the Inuit population. But if an Inuk does not have that gene, it does not somehow make them less Inuit. We can't quantify percentage Inuitness, and doing so is logically unsound. This is because the term "Inuit" doesn't mean its biological correlates. It simply has biological correlates.

To use an example of a personal friend, slightly anonymized: My friend is an Ashkenazi Jew. There is absolutely no uncertainty about this; Jewishness is matrilineal, and their mother was an Ashkenazi Jew, and her mother before her, going back over eight documented generations of family history. But alas - their grandfather was infertile, a fact that was posthumously revealed. Their maternal grandmother had a sperm donor. The sperm donor was not an Ashkenazi Jew. Consequently, can said friend be said to be "only 75% Jewish," having missed the "necessary" genetic correlates? Of course not. By simple matrilineage they are fully an Ashkenazi Jew.

Why are these terms used in medicine, then? Because, put simply, it's the best we can do. Genetic profiling is a useful tool under some limited circumstances, and asking medical subjects their ethnicity is often useful in determining medical correlates. But there is nothing in the gene that says "I am Inuk, I am Ashkenazi," because these ideas are social first, not genetic first.

I don't disagree with this, but this is very not consistent with "ethnicity is not biologically realized", which suffers from the same logical error but in the other direction.

I often wonder how many entrenched culture battles could be ~resolved (at least objectively) by fixing people's cognitive variable types.

In the spirit of randomization and simulation, every culture war debate should be repeated at least 200 times, each with randomly assigned definitions of “justice” and “freedom” drawn from an introductory philosophy textbook. Eating meat is wrong, p = 12/200.

Some day it may make for great training data.

Y-DNA and mtDNA haplogroup are good definitions though. They just don't map exactly to vernacular concepts.

Isn't the problem that the vernacular concepts are what counts and they change depending on time and place?

Indeed, and my patrilineage is similarly defined thanks to my last name.

As a layman who doesn't work with medical studies it always struck me that one of the bits of data that isn't (normally) collected along with everything else is genetic samples of all participants. It should be stored alongside everything else so that if the day comes when genetic testing becomes cheap enough it can be used to provide vastly greater insight into the study's results.

Even something as simple as a few strands of hair sealed in a plastic bag in a filing cabinet somewhere would be better than nothing at all.

That throws out anonymity. I don't see this getting approved, or people signing up for such studies, apart from those who don't care that there genetic data gets collected and stored.

Even if there is no name saved with the genetic sample, the bar for identification is low. The genes are even more identifying than a name after all. Worse, it contains deep information about the person.

I was trawling studies for some issues of my own and sort of independently discovered this many years ago. It's very easy for an intervention to be life saving for 5%, pretty good for 10%, neutral for %84, and to have some horrible effect for %1, and that tends to average out to some combination of "not much effect", "not statistically significant", and depending on that 1% possible "dangerous to everyone". (Although with the way studies are run, there's a certain baseline of "it's super dangerous" you should expect because studies tend to run on the assumption that everything bad that happened during them was the study's fault, even though that's obvious not true. With small sample sizes this can not be effectively "controlled away".) We need some measure that can capture this outcome and not just neuter it away, because I also found there were multiple interventions that would have this pattern out outcome. Yet they would all be individually averaged away and the "official science consensus" was basically "yup, none of these treatments 'work'", resulting in what could be a quite effective treatment plan for some percentage of the population being essentially defeated in detail [1].

What do you mean? They all "work". None of them work for everyone, but that doesn't mean they don't work at all. As the case I was looking at revolved around nutritional deficiencies (brought on by celiac in my case) and their effects on the heart, it is also the case that the downside of the 4 separate interventions if it was wrong was basically nil, as were the costs. What about trying a simple nutritional supplement before we slam someone on beta blockers or some other heavy-duty pharmaceutical? I'm not against the latter on principle or anything, but if there's something simpler that has effectively no downsides (or very, very well-known ones in the cases of things like vitamin K or iron), let's try those first.

I think we've lost a great deal more to this weakness in the "official" scientific study methodology than anyone realizes. On the one hand, p-hacking allows us to "see" things where they don't exist and on the other this massive, massive overuse of "averaging" allows us to blur away real, useful effects if they are only massively helpful for some people but not everybody.

[1]: https://en.wikipedia.org/wiki/Defeat_in_detail

Last I heard, 5 sigma was the standard for genetic studies now. p<0.05 is 1.96 sigma, 5 sigma would be p < 0.0000006.

But even though I'm not happy with NHST (the testing paradigm you describe), in that paradigm it is a valid conclusion for the group the hypothesis was tested on. It has been known for a long, long time that you can't find small, individual effects when testing a group. You need to travel a much harder path for those.

> So the researcher tests if omega 3 effects cardiovascular outcome in these hundred people by adding a lot more fish oil to the diet of these 100 people. Since only one of them really needs it, the P value will be insignificant and everyone will say fish oil does nothing. Yet for that one person it was literally everything.

But... that's not a problem with the use of the p-value, because that's (quite probably) a correct conclusion about the target (unrestricted) population addressed by the study as a whole.

That's a problem with not publishing complete observations, or not reading beyond headline conclusions to come up with future research avenues. That effects which are not significant in a broad population may be significant in a narrow subset (and vice versa) are well-known truths (they are the opposites of the fallacies of division and composition, respectively.)

The real underlying problem is that in your case, genetic variants are not accounted for. As soon as you include these crucial moderating covariates, it‘s absolutely possible to find true effects even for (rather) small samples (one out of a hundred is really to few for any reasonable design unless it‘s longitudinal)

> As soon as you include these crucial moderating covariates

Yes, but this is not usually done.

It is also not easy if you have many potential covariates! Because statistically, you want a complete (explaining all effects) but parsimonious (using as few predictors as possible) model. Yet you by definition don‘t know the true underlying causal structure. So one needs to guess which covariates are useful. There are also no statistical tools that can, given your data, explain whether the model sufficiently explains the causal phenomenon, because statistics cannot tell you about potentially missing confounders.

A cool, interesting, horrible problem to have :)

And it's usually discouraged by regulators because it can lead to p-hacking. I.e., with a good enough choice of control I can get anything down to 5%

The fundamental problem is the lack of embrace of causal inference techniques - i.e., the choice of covariates/confounders is on itself a scientific problem that needs to be handled with love

Anything in health sciences has millions of variants not accounted for, that also interact between themselves so you'd need to account for every combination of them.

From a methods perspective, wouldn't this be more of a statistical power issue (too small of sample size) than a random effect issue? Granted, we do a terrible job discussing statistical power.

This effect would be even more pronounced in a larger sample size. Consider how small a fraction indigenous arctic populations are of the population as a whole. In other words, larger sample sizes would be even worse off in this particular occasion.

But it is more complicated. I have Sami Heritage but it goes back to my great great grandparents. I did not know this until I started digging deeply into my ancestry, but I carry many of the polymorphisms from these people.

So although I look like a typical European Caucasian, my genetics are very untypical of that population. And this also explains my family history of heart diseases and mood disorders which are also non-typical of European Caucasians.

The best way to talk about this is IMO effect heterogeneity. Underlying that you have the causal DAG to consider, but that‘s (a) a lot of effort and (b) epistemologically difficult!

> but that‘s (a) a lot of effort and (b) epistemologically difficult!

I agree, but then all these cheaper, easier studies are useless.

Unfortunately it‘s true that most studies are useless or even harmful (limited to certain disciplines).

Yes. The value is in metastudies and data fusion across more studies.

Even if you did a study with the whole planet there would be no statistical significance since the genetic variation in in FADS genes are still in the minority. (The majority of the world is warm and this is a cold weather/diet adaptation).

In most African populations this Polymorphism does not exist at all. And even in Europeans it is only about 12% of the population.

Watching from the sidelines, I’ve always wondered why everything in the life sciences seems to assume unimodal distributions (that is, typically a normal bell curve).

Multimodal distributions are everywhere, and we are losing key insights by ignoring this. A classic example is the difference in response between men and women to a novel pharmaceutical.

It’s certainly not the case that scientists are not aware of this fact, but there seems to be a strong bias to arrange studies to fit into normal distributions by, for example, being selective about the sample population (test only on men, to avoid complicating variables). That makes pragmatic sense, but I wonder if it perpetuates an implicit bias for ignoring complexity.

It’s because statistical tests are based on the distribution of the statistic, not the data itself. If the central limit holds, this distribution will be a bell curve as you say

Aye, but there are cases where it doesn't hold. Lognormal and power law distributions are awfully similar in samples, but matters on the margin.

For example, checking account balances are far from a normal distribution!

That gets into Simpson's paradox; subpopulations can react differently than the whole.

https://en.wikipedia.org/wiki/Simpson%27s_paradox

Mendielian randomization to the rescue!