The claimed effect size is about a zillion times higher than is plausible
statmodeling.stat.columbia.edu
statmodeling.stat.columbia.edu
During a graduate seminar, someone asked for a show of hands for people who thought a relative risk of 1.10 (i.e. the exposed are 10% more likely to develop a disease than the unexposed) was a big deal.
Very few of the infectious disease epidemiologists raised their hands. We are often used to double-digit effect sizes for our risk factors (HPV and cervical cancer, a particular exposure in a foodborne outbreak, etc.)
The cancer epidemiology folks were like "Yeah, that's a pretty big deal, and worth digging into."
The environmental epidemiology folks were like "Are you kidding? There are going to be lawsuits. Congressional hearings. Maybe arrests."
After I thought about it for a bit though, I had some concerns. I read another paper on a totally different topic where they generated a sort of negative control, and although that paper was interesting too, a problem with it was that it was really easy to argue that the negative control wasn't actually a control, that it was set up in such a way as to inadvertently induce the thing they were trying to eliminate.
I kinda wondered about this paper too. They seem to assume you'll go along with the idea that how you imagine a video game character to act will induce the most extreme effect size you could imagine. But is how you report you'll act even remotely comparable to how you imagine a video game character will react? I'm not really sure. I could imagine some people being like "WTH? I have no idea" and blowing the task off, or seeing the video game character as incoherent and having any kind of behavior consistent with it. People's own behavior can be difficult to predict from what they say, so it seems like a step even further removed to put any weight into what they say a videogame character would do.
I guess it's an interesting idea that deserves more attention, but in my experience, psychological controls are notoriously difficult to implement, which is why you don't see it used more often.
And these are all things estimated from data, using standard, boring statistical methods like logistic regression.
Dynamical systems models, which is what you're talking about, is a whole different field, and also don't inherently predict huge effect sizes - the last non-COVID one I was working on, for example, was estimating fairly small effect sizes.
For example, cervical cancer without HPV infection is extremely rare, but non-zero. Or someone at the party who didn't eat the potato salad could still come down sick by another pathway.
On the other hand, for things like environmental exposures, you're often looking at very small increases in risk over baseline. For example, your increased risk of asthma with a higher increase in particulate matter in the air.
(The example the parent gives is "HPV and cervical cancer" - it's not necessary to have HPV to get the latter, so those not exposed to HPV don't get a 0% chance of that disease).
E.g. say a study finds that if you're exposed to VirusX (or ShouldNotHaveBeenApprovedByFDADrugX, as this applies beyond viruses) you have 10% more chances to develop a heart arythmia than without being exposed to it (which, if I'm not mistaken, the latter should be the same as saying: "as compared to the control group").
One is non zero, and the other is non zero + X.
I once mapped a modulator of lifespan in a small family of mice precisely to a chunk of chromosome 2, but using only 24 BXD family members (https://pubmed.ncbi.nlm.nih.gov/23698443/). The placement of this locus is solid, but the effect size is inflated about 3 to 5X.
In recent work in mice we know the effect sizes for single longevity loci max-out at about 30 days per gene/allele, but in the small study above the effect size was about 100+ days. Eye opener for me.
Beavis effect evaporates with much larger sample size. In our current research on the genetics of longevity in mice we are up to 6000+ animals and have about a dozen significant hits that collectively explain roughly 125 days of life span difference. Should be in bioRxiv in next month or two Arends DA et al).
> we were interested in the question of how many participants a typical between-subjects psychology study needs to have an 80% chance to detect a true effect.
> Turns out you need 47 participants per cell to detect that people who like eggs eat egg salad more often than those who dislike eggs.
This idea would seem to suggest special restraint when negatively interpreting a lot of A/B test replication. It's common to get a good result with a small sample, retest with a much larger audience and get a less-impressive result. If I understand, that outcome should not necessarily be discouraging for the success of the experiment. No?
Also weakly informative priors, which have regularizing effects.
Actually these two things are a big focus area of Stan, which is the probabilistic programming language built by the group which writes that blog.
> That is the type of effects that has a Cohen’s d of around 2: Tautologies.
I thought the counterargument to this finding was that judges are reasonably good at predicting which cases are going to be slamdunks versus which are going to drag on. What I don't recall is if the judge is involved in sorting the cases for the day, or just in looking at the next case and saying, "You know what, this is probably going to drag on for two hours at the end of which I'm going to say "guilty" or "denied" so I should just call lunch now.
Which would make the case immediately before lunch a simple yes/no, and the one after a complex 'hell no' a statistically significant percent of the time. Tautology. Or maybe judges don't like giving bad news on an empty stomach.
You are entitled to due process even if no bookie would take odds on the outcome of that process. At the end you're going to jail unless your lawyer really surprises me, but you'll have had your day in court and hopefully that will help you appreciate just how badly you done fucked up. But you are not entitled to plead your case while I'm hungry and need to use the bathroom. Court adjourned until 1 pm.
I generally assume that more intelligent people than me would have stumbled onto any connection that obvious. So, I always take it as a warning sign that I've messed up a parameter or put a proxy variable into my training data if I see things reaching that level of effect.
The first measurement of the speed of light was too low. Subsequent experimenters assumed that when their measurement differed from the previous measurement by too much, then their experiment was wrong. So they fiddled with it until it wasn't too far off the previous value, and then published.
EDIT: it was about the charge on an electron, not the speed of light, my mistake: https://scipython.com/blog/measurements-of-the-electron-char...
If you assume that it's your calculation errors that lead to observed discrepancies (in eg. speed of light), you won't be able to measure speed of light until you can get those errors below the observation threshold.
If you don't even assume that you can get those errors below the threshold, you won't try and you will live happily in belief that the speed of light is, well, infinite.
Bookmarked :)
To illustrate: suppose we've found a posterior distribution for effect size, and we've found that 95% of the posterior probability mass is >= 0. It might be a broad flat distribution (huge MAP effect size, massive uncertainty about effect size), or it might be a tight distribution (small MAP effect size, small uncertainty about effect size). There's nothing intrinsically wrong or suspicious about either of these two cases.
The headline of the linked article has the words "higher than is plausible". In other words, the premise of the article is that there is some other evidence which tells us what effect size is plausible. We could incorporate that evidence by adjusting our prior, which will obviously push the posterior distribution closer to what prior belief says it should be. Alternatively we could incorporate that evidence as a subsequent Bayesian conditioning step, i.e. treat the other evidence as further observations. The magic of Bayes's rule means that both of these approaches give the same answer. (The frequentist approach is pretty much like the latter -- it says "here's the result of my experiment, and I'll leave the reader to incorporate it with their own prior beliefs, in a meta-analysis." There's nothing intrinsically wrong with this.)
The article says "I find it frustrating when researchers don't think about their effect sizes." That's the wrong conclusion. The proper conclusion is (1) always report confidence intervals for your effect sizes, (2) don't pay any attention to a statement about effect sizes unless it comes with confidence / credible intervals -- and so the article's "painful email exchange" is missing the point, (3) when you report your results, make sure there's enough information for the reader to incorporate it with their own priors -- and the whole point of confidence intervals is to let us do this.
General rule of thumb from my experience though is that if the author is resorting to p values to justify their findings they're either a fool or an idiot, or both... (I'm not criticizing p values but they're not the only metric of statistical worth, nor are they an accurate reflection of complex systems)
Be extremely wary of any one who just says 'It works' to justify an action and gives no context.
'It works' always has a ton of context. How much does it work? What situations does it work? What is the sampling error? Is the tradeoff for that improvement worth it?
There was also this one by someone else: https://news.ycombinator.com/item?id=26684519 https://news.ycombinator.com/item?id=21546850
But ... Obligatory xkcd. https://xkcd.com/2400/ If you have to use statistics at all, then your data likely isn't good enough to be repeatable. I'm sorry about this. I wish the world was easier to measure.
http://daniellakens.blogspot.com/2017/07/impossibly-hungry-j...