Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

Well, there was that whole dotcom boom and a lot of things changed for computers & the internet which led to researchers being able to share more information, use more powerful computer techniques, etc.

Also I had not heard of Kevin Drum before this, and had a positive view of Mother Jones. I'm left with a poor impression of Kevin Drum and a hit to Mother Jones' reputation after reading this.

Literature reviews in meta-analyses are usually conducted like this, with specified lists of search keywords and flow charts of inclusion criteria.

There are explicit exploratory methods you can use to carry out research and that is very fine, but most research today is done using an inference model, where you formulate a hypothesis and use data to corroborate it.

That is one of the premises of big data that most people do not get: It enables you to do research explicitly exploratory again, while still being rigorous and somehow falsifiable..

I mean, can't this be attributed to other things too? (The way people live has changed in the past 20 years...)

I'd rather see some studies to show if/why those things have spiked rather than disavowing attempts to help it.

What else are we supposed to do?

Recent US outcomes: https://jamanetwork.com/journals/jamacardiology/fullarticle/...

US & New York State outcomes 1980-2007, Figure 2: https://www.health.ny.gov/diseases/cardiovascular/heart_dise...

I haven't hit the major mortality sources, but these bits match everything else I've heard.

I'd also like to know the basis for asserting that the health care industry doing more harm than good.

Clinical trials are not basic research, though they rely on a great body of basic research, and are themselves experiments. Requiring that clinical studies have clearly defined end points serves the purpose of ensuring that the results are robust and well understood. This is important because the end goal of a clinical trial is developing a treatment of some sort that will go into many humans, and mistakes can be very harmful.

Choosing an appropriate endpoint for a clinical trial can also be very challenging. Say for example you're trying to advance a drug candidate for treating cardiovascular disease. You have many choices for how to measure that - chest pain, resting heartrate, cholesterol etc. It's very much possible that your drug does improve cardiovascular health, but your trial can fail because you chose the wrong endpoint.

When I perform an experiment, I do so with a hypothesis which is sometimes proven right, and sometimes proven wrong. Either way, the results can be interesting and form pieces of the large puzzle that is a scientific study. Further, questions that are addressed in a scientific study are generally going to be more broad than those addressed in a clinical trial - we don't really do exploratory clinical trials, but exploratory research is a very important part of science. For these sorts of studies, it is often difficult or impossible to have a well defined endpoint such as is required in a clinical trial. One example for you: I hypothesize that enzyme X has function Y. During my studies, I discover that enzyme X actually has function W! If the evidence I present for enzyme X having function W is solid, my results can still be great and useful, even if my initial hypothesis was wrong.

"A reasonable takeaway from this article is that any one scientific study may contain flaws, or show bias, and therefore there is a possibility for it to be confirmed, improved, or disproven in the future. As a result, it is better to take early results with a grain of salt until additional supporting evidence is found, than to take them as gospel.

"It should be noted that these results are due to flaws in the scientific establishment, flaws in the human application of science, and flaws in the humans themselves, but not the scientific method itself. The scientific method itself is sound and can be trusted. You use it every day to see whether the water in your shower is too hot or cold, or whether it is raining outside."

Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing?

Assuming that the effects are legitimate is exactly the problem. You are right, if we somehow know that the effects found are real and reproducible, then all is good. The problem is that we almost never know this. Presumably what you mean is "If the results are reproducible, what's the harm?". I'd agree with this, but the problem is how to know ahead of time that the results are going to be reproducible.

Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result?

If they did the statistics correctly (accounting for the multiple inferences, all assumptions about iid data met, no biased dropouts, everything else aboveboard) and got a really solid result, then yes, it's theoretically likely that the results would be reproducible. The problem is that they almost certainly didn't do the statistics correctly, and intentionally or not they probably violated a lot assumptions. In too many cases, even the main line conclusions can't be replicated. It's rarely "safe" to assume that an effect is real until it's actually been replicated, and almost never safe to make this assumption for result obtained by sifting the data after the fact looking for correlations.

I'm having a hard time seeing the problem with taking an exploratory approach and just testing placebo vs. some treatment and reporting whatever you find.

There are ways to do it this way, but it usually takes larger sample sizes than are available. It also requires "bespoke" statistics that are easy to get wrong. In practice, it's usually better to use the incidental "results" as idea generators for future experiments, rather than assuming that the findings are real and don't require further testing.

Indeed. But this doesn't disqualify looking for "something" as a valid and useful method of research to be a stage of the whole research chain. That ought to be allowed although research papers produced this way should make this clear.

That still seems a good first step. It may also be curious to find out what were the things that have actually lead to the effect if not the one we were checking. E.g. it may happen that it the substance researched only affects subjects with particular features or that something else they were doing has produced the result on itself and knowing any of these still seems useful (some really important discoveries were made this way AFAIK).

I assume that taruz is saying that journalists should be required to say what they're investigating before publishing an investigative report. Right now, they start investigating, and if there's something outrageous (even if it wasn't what they were initially looking for), they publish. Sometimes they even skip the "start investigating" part and just put up a tip-line for anyone who has a beef to get a story out there.

This is great for manufacturing outrage and hence clicks, but it gives the public a hugely skewed perspective on how the world is. Imagine that 0.01% (1 in 10,000) of all peoples' actions are outrageous and will piss off a large portion of planet. Most people, by those numbers, would say that the majority of folks are decent, law-abiding citizens. Now imagine that a news outlet is allowed to freely go over someone's life, and they end up evaluating 1000 actions. There's a 10% chance that they'll find something outrageous. Now imagine that 10 such reporters do this to 10 people, and if any one of them finds something publishable (= outrageous), they go to press. Suddenly there's a 64% chance that one of them will find something, and you've likely got your news cycle for the day.

With the millions of people looking for something bad that a tip-line can generate, outrage is virtually assured. And that's where journalism is today. The world isn't actually a worse place than it was in 1980; in fact, by most metrics it's significantly better. But we've increased the amount of unpleasantness that people can be exposed to by 3-4 orders of magnitude and then implemented a selection filter that ensures that only the worst stories go viral. Of course we get only bad news; that's all that's economically viable, and we have such a large sample size that we can surely find it.