Medical Science Has a Data Problem
newsweek.com
newsweek.com
- The panel is stacked with visiting researchers working for the most senior researcher on the panel.
- The most senior experts on this panel recommend treating with a particular drug. It turns out that these experts also happen to receive money from these drug companies.
- These researchers then use the results of this panel when applying for NIH grants as proof that this is where the treatment is heading AND THEN, next time the panel happens, also use this research as proof that this is where the treatment is heading. Talk about self interest and circular logic.
What is incredibly sad about this is that the results of consensus panels are then quoted by doctors every day and used to lead their treatment of patients.
How does this practice keep us safer?
The thing about medical studies though is that only the ones the pharma companies want to get published make it into journals, and there is no requirement that what they are saying in the journal articles matches the clinical trial data at all. And the FDA doesn't publish any of the raw clinical trial data, so you'll never know.
This is especially problematic since pharma companies only need two studies showing a drug plausibly works to get approval, regardless of how many trials fail to show any benefit. So usually what you see is one or two journal articles showing a drug works, which may or may not correspond to the findings of one or two clinical trials, and the rest of the trial data is buried forever. And then other authors publish metareviews based on this completely inaccurate data.
Essentially journal articles should only be ever be viewed as advertisements for the pharma industry, and should no way ever be taken as science. The one exception is the largescale trials funded by the NIH.
However, it is true that most of the controversial, blockbuster drugs that are discussed in the original article were approved long before these requirements came into effect.
- A substantial percentage of trials haven't reported their results within the required timeframe.
- The number of fines the FDA has levied for non-compliance is zero.
- The data isn't actually published on clinicaltrials.gov, only a brief summary of the results which doesn't really tell you anything.
- As you say, the requirements aren't retroactive.
- There are several large loopholes, e.g. phase IV studies aren't required to be registered.
There's a fundamental lower bound to the number of data-points you need to make in order to achieve statistical significance. This lower bound grows exponentially in the number of dimensions, particularly if you cannot make the simplifying assumption that the dimensions are statistically independent (and biological systems are often coupled in strange ways). Randomization is a good way of achieving that lower bound, but it cannot defeat it.
Yes, and then there is the influence of the drug makers too.
The other point is complexity. The number of variables needed to create a useful model of a jet engine is a lot smaller than the number needed to model an organic system. This means that today we have useful models of jet engines and we can reliably test something with simulations - we may generate loads of valid data and easily apply "top-notch" statistics. The same, unfortunately, is not true for medicine.
I think it's becoming clearer each day that many drug side effects, and general drug efficacy is effected by the genetics of the drug taker. Without being able to tease out these genetic effects, blindly aiming for broad population safety or efficacy levels is always going to be troublesome.
This is separate from the data publication issue that the article focuses on.
Not just expensive but also potentially unethical.
Consider this: post-exposure prophylaxis for HIV. It's used both for sexual and non-sexual exposure to the HIV virus. It's a month-long series of antiretrovirals (the exact protocols used differ). It's effective if you start taking it within 72 hours of exposure.
The drugs themselves have been tested for use by people who are HIV+, but nobody has done clinical trials on using them as post-exposure prophylaxis. We know it works, but we don't know how well because testing would be unethical. You'd have to take half the people who are exposed and give them placebo and put them at risk of getting HIV.
At a certain point, you have to throw the dice and say "given the evidence we have, this is what we are going to do".
(The fact that such judgement calls are being distorted by evidence being intentionally withheld by drug companies is disgraceful though.)
Confidence of effectiveness is very easy to get - you just need to find an effective drug. e.g. when antibiotics were first tried - they are massively effective compared to previous treatments - you get high confidence of effectiveness from just a small sample size.
A reason it is hard for new drugs to get high confidence is because they are only marginally better than existing drugs.
But new therapies have to compete with existing therapies in the market anyway. If a new therapy can't outperform existing ones, what actual value does it have? Perhaps drug companies should think about solving problems that aren't already solved rather than bemoaning their inability to reap massive profits while reinventing the wheel or pushing minor formulation tweaks to renew their patents.
Take any drug, test in on millions of patients, here is your six sigma. What problem did you solve?
Your comment shows me, that not much has changed.
If there is something different biologically different about the 1,000th person, could it be tested for?
Drugs act on proteins which are little machines in the body, the instructions of which are encoded in our genome. The slight variations in our genetic code leads to some proteins having slight variations in their structure (and their function). Most of the time this is harmful, but sometimes it causes them to react differently when they interact with a drug or with each other. The manifestation of this can sometimes be harmful. Understanding "pharmacogenomic" effects will help us mitigate these side-effects going forward.
There are far more combinations and variations of the above variables and sub-variables than can be tested for in clinical trials. This fact means that some obscure combination of these variables may result in unintended negative (or positive, less frequently) consequences upon consumption.
Theoretically, you could test for differences in the 1000th person-- with today's technology, it would be a test which described the person's genome. Once scientists had that genomic data in hand, they'd certainly find a profile of alleles which could feasibly describe why a negative reaction occurred.
Importantly, everyone's genome is unique, meaning that a correlation between genomes and negative drug reactions must be exceptionally rigorous in order to be believable. Currently, this correlation poses a problem, as our ability to quantify and understand genomes is far ahead of the other dimensions which I mentioned at the start of my post. As our understanding of the microbiome, enteric connectome, and epigenetics increases, it'll be more possible to predict adverse drug reactions beforehand, assuming the patients in question have been profiled.
That said, an important point that hasn't been made here yet is that "alternative" treatments to the treatments developed by medical science often have a much worse data problem, because no one is gathering data about the alternative treatments for safety or for effectiveness in the first place. In the United States, "natural supplements" are regulated in a way that exempts billions of dollars of annual revenue from any review about whether the supplements do any good at all.[2] Chiropractic manipulation can kill, but chiropractors don't look into that issue carefully.[3] What you as a person who desires to stay healthy for many years have to do is turn in the first instance to medicine that works[4] and to continually ask your doctors for explanations of what they are doing and what the rationale is for what they are doing. Patients thinking more will encourage physicians to think more. Voters telling governments to regulate treatments more rationally will also help, as will voters supporting more funding of basic medical research.
[1] https://med.stanford.edu/profiles/john-ioannidis?tab=publica...
[2] "Dietary supplement industry says “no” to more information for consumers (again)" http://www.sciencebasedmedicine.org/big-supp-resists-giving-...
[3] "Stroke Death from Chiropractic Neck Manipulation" http://www.sciencebasedmedicine.org/stroke-death-from-chirop...
[4] "There is no such thing as alternative medicine. There is only medicine that works and medicine that doesn't." http://books.google.com/books?id=zOh0t3QFXdoC&dq=There+is+no...