When doctors, psychologists, and drug makers can't rely on each other's research | Hacker News Reader

When doctors, psychologists, and drug makers can't rely on each other's research

reason.com

67 pointsnkurz26 comments

>We are living in a time of technological marvels, with advances like CRISPR gene-editing being used to bring back extinct mammoths; lithium-air batteries that store 10 times more energy than conventional lithium-ion batteries; mitochondrial transfers that create healthy babies who have three genetic parents; Ebola vaccines that are nearly 100 percent effective; and cars that drive themselves.

This line was sort of ironic, since we don't really have those things. Mammoths are still extinct, lithium-air batteries aren't yet practical, three parent babies aren't legally deemed as safe in most countries, and cars can only drive themselves in limited ideal conditions (and those are only research vehicles!). Was that statement was put in as another example of how you can't believe what you read and it went over my head?

I took that line as saying that it was remarkable that those things are happening at all rather than saying they are safe, practical and fully functional.

I mean I'm impressed that they have actually inserted mammoth DNA in to elephant cells even if Jurassic Park will have to wait a bit. (http://www.livescience.com/50275-bringing-back-woolly-mammot...)

> The model he has in mind is the e-print distribution platform arXive.

> Other disciplines are emulating the arXive pre-print model

> ...something like a gigantic open-source version of arXive for all scientific research

That's not how you spell arXiv.

What it would take a resarch-funding organization to issue grants for specifically replicating results of existing papers? Why don't they do it? May be they do and I'm just unaware?

They don't because it's not considered cool.

There are organizations that are attempting to police published research...one such is Retraction Watch...

It's a 501(c)3, thus a not-for profit...not sure how much grant funding they receive, but they do actively solicit donations from the public...

I think you'd be surprised at the number of public research papers, including some that end up being sources for articles in well-known science journals, that are retracted each year...

A list of papers retracted, which is continually updated, can be found at their site:

http://retractionwatch.com/

I like this summary of current research on how to do research better. A checklist of suggestions for psychology research journal editors[1] gives a different set of links to current research and current best practice, and is well worth a read too.

[1] https://jcoynester.wordpress.com/2016/01/21/ten-suggestions-...

This is largely predicted by Daniel Kahneman and Nassim Taleb, would love to see them cited more often.

"HARKing" is a great coinage I had not seen before.

Ladies and gentlemen, Sturgeon's law at work.

Why are people so obsessed with publication bias and statistical power? Those are just two of the dozens of reasons why published research is unreliable, and it's not even clear that they're the most important ones.

It seems like the skeptic community just randomly glommed onto those issues in the 90s or something and haven't updated their worldview since.

What are the other, more important reasons that research is unreliable?

I have a list of them going here, but it's definitely not comprehensive either:

http://www.alexkrupp.com/Citevault.html#pharmaceuticals

Even if you look just at statistical analysis, there are dozens of different issues. E.g. I seem to remember some articles about how basically 100% of research involving brain scans is incorrect, because of a mix of statistical problems and also issues related to the imaging software.

This section is particularly useful to me:

http://www.alexkrupp.com/Citevault.html#mentalhealth

Thanks!

A big problem are the perverse incentives researchers are facing. The only thing that matters is publishing in prestigious journals, otherwise your career is pretty much over (no real chance to get tenure). Universities often don't even check the publications, and researchers usually omit the actual title of the publication and only keep the name of the journal. Moreover, if you want to get tenure you need to work on "hot" topics (as in CRISPR at the moment or cancer in general), meaning you don't necessarily spend too much time on one subject. I'm sure researchers are often aware that the research results are not ready for publication yet, but it's either publish or perish, and you usually get away with publishing sub-par results because no one really follows up on them either. Cancer and Alzheimer are two fields were billions have been wasted by research grants and pharma companies because the initially promising drug targets for example could not really be validated. I'm afraid this will stay that way as long as the career dilemma researchers are facing does not change.

I suppose I haven't read enough about publication bias to get as annoyed by it as you seem. TFA offers a pretty convincing argument that publication bias is bad enough to doom most current scientific efforts to irrelevance. That sounds like something we should talk about? Or we can talk about more important reasons why published research is unreliable, if you care to mention them.

What is "the skeptic community"? Is Reason part of it? I'm a bit skeptical that's even a thing...

Statistical power is mentioned once in TFA and then forgotten. Amusingly enough, it gets the relevance of statistical power for the subject under discussion backwards. That graf might actually have been added by an "obsessed" editor.

Reason is generally a libertarian publication and website.

For the Skeptic Community go to r/skeptic. They're related to the Atheist Community.

So by "skeptic community" you actually mean "reddit skeptic community?"

There are skeptic conferences, skeptic magazines, skeptic books, skeptic authors, skeptic web forums, skeptic TV shows, etc. I think this easily meets pretty much any definition of community.

This seems to be focused toward critiques of religion rather than of science? I don't see the connection.

In double-blind tests, one deliberately changes only one variable and compares/observes two groups where individual do not know whether they take a new drug or a placebo. If the drug has a positive effect, the double-blind test is an accepted method of proof, but unfortunately this limits our options severely. If cancer can be cured by changing _two_ variables, we will never find a cure. NEVER! Remember that we already do research for 50+ years and have spent trillions, so I tend to think that Einstein was right: it is insane to repeat the same experiments and to expect a different outcome.

Let's jump to Alzheimer, a feared disease with also a lot of research and not so good drugs. Dr Bredesen has made a protocol with 35 variables to cure Alzheimer. And in his first test, he reversed Alzheimer in 9 out of 10 patients. The scientific purists say that there is no double-blind study, so no proof. I am convinced that Dr Bredesen in on the right path, not only because of the results but also because of his reasoning. The treatment has no drug, but a health optimization in all possible ways. And then the body heals itself in 4 months. Please do not comment with "that cannot be true" unless you proof that it cannot be true (I do not belief that you can provide proof).

So the state of current research methods is weak. Medical research only focuses on one variable, one drug, that will cure a disease. The results of this way of doing medical research are VERY disappointing. Trillions are spent and no drugs that cure cancer, AIDS or Alzheimer have been found. Dr Bredesen has chosen a different path and I support him.

So if you have an X variable cure, that means that an X-1 variable cure is not enough, right? I'm not very familiar with the nuances of the scientific process, but I would think that you could still run tests on one variable at a time.

Just turn all but one variable "on", make that your control group. Turn all variables on, that's your experiment group. Now repeat the process isolating each variable, determining whether they're all in fact necessary for the cure.

Unfortunately you have a fundamental misunderstanding of health experimental design. Double-blind means that neither the participants nor the experimenters know which participant is allocated to which group. It doesn't say anything about how many groups there are or what is done to each group. I haven't heard of Dr Bredesen, but they could certainly run a double-blind study to validate their treatment regime. Treatments that haven't been validated in this way have a high risk of being ineffective but appearing effective due to experimenter error, placebo effect, demand characteristics, etc. Requiring convincing evidence in the form of a double-blind trial isn't purism, it's sensible caution based on experience.

edit: you're also on shaky ground statistically. There's a rule of thumb that if interactions between several variables are present in a treatment effect, 70% of that effect will show up on the individual variables (as main effects). It's very unlikely that there exists a treatment where all the effect is on a 35-variable interaction and none of the effect is visible when looking at each variable singly.

> It's very unlikely that there exists a treatment where all the effect is on a 35-variable interaction and none of the effect is visible when looking at each variable singly.

Maybe something about genetic modifications where 35 different point modifications to the genome have to happen in order to allow synthesis of a working protein? If you measured the effect of each point modification, you might measure it as zero.

I'm analogizing from trying to do statistical tests of whether a password is correct. At least in systems that aren't vulnerable to timing attacks, you can't just test part of the password and see if you get a little bit more access to the system as a result. :-)

You need to read my comment again. I never stated nor think what you wrongly assume.

Your reasoning about the work of Dr Bredesen shows that you a re ignoring the facts. reversal of Alzheimer in 9 of 10 patients is a very significant results since nobody has ever reversed Alzheimer. The fact that the sample was small is only a reason to do a followup study with a larger number of patients, which is exactly what he is doing.

The desire to reduce protocols to only a single variable is based on the wrong assumption that treatments can be successful with only one variable changed and excludes all treatments where more variables play a role.

Treatment X, may contain drug multiple drugs and be treated as a single variable. Further, plenty of statistical tests work with multiple groups not just A vs. B.

As to a cancer cure, we gotten ridiculous progress. The problem is cancer is effectively evolution and lacks a single cause.

Reply on news.ycombinator.com