Hacker News Reader

Comment by sandoooo | Hacker News Reader

How do we prove that they didn't just look at mortality in all diseases and for all restaurants, and cherry-picked stroke-vs-ramen because it has the best significance?

I am not accusing them of anything - I have no horse in this race. What I am curious about is: how do you prove that you haven't p-hacked, if you are going to do a study like this?

Sure, you can pre-register your methods, but presumably you can get the data and do some data crunching on it way before you have to pre-register. How do you prove that your results are legit in the age of reproduction crisis?

View on HN

> How do you prove that your results are legit in the age of reproduction crisis?

You... try to reproduce them. The "reproduction crisis" isn't a failure of the scientific method, it's just pointing out that we've picked all the low hanging fruit (i.e. highly significant results) in science while increasing the amount of science done by huge amounts. So there's a lot of junk. Possibly (hopefully! I love ramen) including this paper.

But nothing has changed with how you prove your results are legit: do a similar study in a different location trying to find the same effect.

Good faith is required to accept any type of reporting. Otherwise, your question can always be extended to "How do we know that they didn't just make it all up?"

The researchers did not report conducting more comparisons and they did not adjust for this in their analysis, so we have to assume that they are telling the truth.

Of course, there's always the chance that they are being deceitful, so we have to reserve some skepticism and take into consideration related studies before forming any firm conclusions.

In this particular case, the correlations that they found have quite strong effect sizes with p < .001. Figures 2 and 3 are quite striking. These results are also supported by previous research. That doesn't mean the matter is settled by any means. It's just another piece in the puzzle. But there is no indication whatsoever that they are p-hacking or going on a fishing expedition.

I don't think you have to assume good faith at all in these situations.

If someone comes up with a correlation of X statistic with Y statistic in a fashion seemingly coming from nowhere, they either say why they started studying the correlation or show the correlation is incredibly strong.

I think that's pretty much what "extraordinary claims require extraordinary evidence" winds-up meaning.

The correlation is so large that I find it literally incredible.

Could ramen restaurant prevalence really explain a quarter to a third of the variance in mortality (which is what r=0.5 to 0.6 implies)? If so, the damn things are an outright menace.

And why is it so different from fast food, which is equally unhealthy but supposedly makes people live longer?

What is the connection between a correlation and an explanation? I see none. Zero. Nada.

It is pretty common to interpret r^2 as the fraction of variance in one variable “explained by” the other. Here, ramen restaurant prevalence and mortality are reported to be correlated at r=0.5 to r=0.6, so r^2=0.25 to 0.36

Obviously, “explained” here doesn’t indicate something conceptual or causal—-it’s more like “accounted for”—-but the phrasing is probably too well established to change now.

And that's sort of my point...It doesn't seem plausible to me that ramen would be that deadly, so I suspect there's some other unmodelled factor that drives both restaurant prevalence and mortality. For example, maybe ramen is more popular in poorer/more rural areas, which have worse medical care/outcomes.

The authors discuss this in the paper's Discussion.

The prevalence of ramen restaurants is suggested as an indicator of general food preferences in the regions studied. Regions that have a higher prevalence of ramen restaurants also tend to have a higher salt intake in general. This can be seen by the increased use of soy sauce and miso compared to other regions in Japan (from personal experience, these regions are famous throughout Japan for their high-salt diets).

So I'd say that the authors are very careful to clarify that they are not implying any sort of causal relationship between ramen restaurants and stroke mortality, and I don't think anyone reading the paper would assume that either.

I did see that, and maybe Japanese regional cuisine varies a lot more than I'm imagining. However, I'm still not convinced that something higher-level is going on.

For example, I think you could find a similar correlation between (say) soul food restaurants and diabetes prevalence in the US. Soul food tends to be starchy and fatty itself; you could also make a similar argument that the number of soul food restaurants is a proxy for food preferences in each region.

However, soul food is especially popular in African-American communities. There are genetic factors that predispose African-Americans to diabetes. Soul food restaurants are probably more common in poorer areas. Diabetes is also linked to poverty: partly through diet, but also through things like access to preventative healthcare and management of pre-diabetes. Perhaps there still is some residual effect of the restaurants themselves, but it would be much weaker after adjusting for other factors.

You don't at the level of this study. It's useful to suggest this is something that might be explored in a better, more clinical, setting.

The news might pick up and run with this, which is unfortunate, but so long as you don't view this as anything more than a suggestion of a relationship, you should be fine.

I guess the only reliable way is to have an independent group reproduce it using different data and publish their result.

Whenever theres something interesting shown in natural sciences you quickly hear about replication efforts being able or unable to replicate the results. In most other sciences there doesn't seem to be much glory in replication

Reply on news.ycombinator.com