Hacker News Reader

Comment by hombre_fatal | Hacker News Reader

Why not critique the methodology instead of washing your hands of the paper?

Corporations and industry are the only ones willing to sponsor all sorts of studies. Can you at least show us some sort of historical convergence rate of industry vs non-industry funded studies? Is it 10%? 90%? If you have such a fatal knee-jerk response just seeing the funding, you must be able to do that, surely.

Frankly, I'd expect industry to simply discard studies that don't support their product (null or negative result) and publish the studies that do promote their products. Either way, I'm left reading the methodology no matter who publishes a study.

View on HN

The problem is, reading the methodology can only give you a negative signal or a null signal. If the methodology looks good, that still doesn't speak to the correctness of the study's conclusion, for exactly this reason:

> Frankly, I'd expect industry to simply discard studies that don't support their product (null or negative result) and publish the studies that do promote their products.

There could have been a hundred attempts at this study before, all with a negative result, and none of them got published because they didn't have the result industry wanted. We simply don't know. But theoretically, this is something peer review should be able to fix: Researchers without industry support should be able to get the same results if they run the same methodology. Until then, being skeptical of the result seen, even if the methodology is sound, is completely reasonable.

All of the responses to my comment just explain the issue with having a single study available on a topic. You have the same exact issues with independent grants and researchers who want to uncover some real results and not just waste their time with null results, or they want continued funding.

You could level this criticism at any study much simpler. "Oh, the researchers just wanted this to be true." Why even bother singling out industry interest?

It's why meta-analyses are at the top of the evidence hierarchy: they look at multiple studies from varied sources.

But I'd still like to know how much industry funded research converges with non-industry-funded findings. Else you're doing the equivalent of dismissing, say, observational research even though it converges with RCTs 60%+ of the time which I think definitely takes some wind out of the sails of the "observational research is BS" knee-jerk.

The issue other than the one of visibility that you pointed out (meaning the study you requested can’t exist) is not just the paper but the potential for continuation of funding.

As you said, corporations have lots of money, and people who find results that support the corporation’s desired outcomes often get relationships and additional funding. Those who don’t, don’t get relationships and additional funding.

This has at least two consequences:

1) There is a strong and ongoing financial incentive to fudge results in these studies.

2) Those who refuse to do so may not get funds that would allow them to continue in their field at all.

Most of such funded studies would have been better to not have been done at all. They don't just add noise, they add negative signal.

And the methodology means very little. There are tons of ways to cheat and skew the results, while the description of the methodology used remains the same.

Reply on news.ycombinator.com