Why Are Clinical Trials So Complicated?
blogs.sciencemag.org
blogs.sciencemag.org
I also don't think people should harsh on the FDA because of their strictness. There's a lot of gaming attempted in dealing with them which makes them harder to work with, but in my experience they've been pretty decent to deal with. Of course we had a policy of no attempting to game the system.
And then there's the more malicious "gaming", fraud! Clinical research is obviously heavily incentivized toward positive trial outcomes, but for each site the act of just enrolling patients is incentivized, so you need to be damn sure the patient data (and the patients themselves!) in your study are real. And being sure is not easy or cheap.
For fun, here's the FDA's list of people who have been blacklisted from performing clinical research: https://www.fda.gov/inspections-compliance-enforcement-and-c...
Their associated "Volume Page" lists (in excruciating bureaucratese) their "crimes", which may be unrelated illegal narcotics peddling (oops) or falsifying patient data.
I agree that the shift to biomarker endpoints has made the process a bit more aggressive (in probably a good way) but has also lead to all sorts of opportunities for gaming you way to pointless success...but then you have something in the market with something you can talk about!
What I've appreciated about the FDA is their scientific focus. The FDA might say "You need to do X", but if you can come back and convince them with a scientific argument that you don't have to, they will listen to you!
And as a consumer I'm very happy that if I get prescribed a drug it's been thoroughly evaluated for safety and efficacy.
I would very much struggle to pick on a point in time up to now where a society picking 'do things safely' has developed a real advantage over a society that went with 'learn quickly'. The learn quickly people make truly horrific mistakes, but then have an opportunity to learn from them and settle into much more sophisticated equilibrium where they achieve better future results in perpetuity.
Government regulation makes things safer by banning unsafe practices. We don't see the damage this does because stuff that doesn't happen never gets seen. However, it is worth holding on to the idea that it just isn't the government's job to keep someone safe when the person involved doesn't care. People should be allowed to buy and treat themselves with potentially unsafe or ineffective drugs if they understand the risks and have read up on cases like, eg, Thalidomide. There will be benefits from learning from the mistakes and since it isn't possible to live a risk-free life the government shouldn't be trying to enforce it. A strategy of education and information is great. A strategy of prohibition and red tape is actually big-picture risky in itself with a a counterfactual point of view where things are experimented with widely.
Thalidomide didn’t have a sticker that said “will cause birth defects in your unborn child”.
Medical science has produced untold numbers of what would be classified as miracles 70 years ago. I can name a half dozen people I’m personally acquainted with who live today because of effective, safe treatments.
As the disgusting affair around COVID has demonstrated, hucksters and idiots will happily push unsafe, ineffective treatments for a variety of self-serving reasons. Or if that’s too controversial, how many needlessly suffered from application of junk cures like bleeding or mercury?
How do you do it for the people in a clinical trial? Somebody has to be the first.
> Medical science has produced untold numbers of what would be classified as miracles 70 years ago. I can name a half dozen people I’m personally acquainted with who live today because of effective, safe treatments.
The argument is not that no progress is made using more conservative methods, it's that there is a level of "safety" where the additional progress not made would have saved more lives than would be lost to somewhat less conservative methods.
> As the disgusting affair around COVID has demonstrated, hucksters and idiots will happily push unsafe, ineffective treatments for a variety of self-serving reasons.
Meanwhile the bureaucratic red tape around initial testing allowed the virus to spread as far as it did in the early days, dramatically increasing the scope of the problem and costing many lives.
> Meanwhile the bureaucratic red tape around initial testing allowed the virus to spread
I see this lazy argument all the time. A test that has not been characterized and qualified produces unknown results (what percentage are false positive? False negative? Does that improve if you tweak the protocol?). If your test produces unknown results they are simply noise, and increasing the rate at which you produce noise cannot improve your judgement.
That doesn't flip it around at all, it's still the same problem. There is a trade off between costing lives by being too reckless and costing lives by being too conservative. Your argument in fact exacerbates the problem of being too conservative -- of being too far away from the optimal trade off in either direction -- because it applies just as much to the lives lost to slower progress.
And banning lead or similar is a completely different situation because the data is in on lead. It's not prohibited because we don't know if it's bad for you or not, it's prohibited because we do know that it's very bad for you with a high degree of confidence. It's obvious what to do with things that are well known, the question is what to do with things that are not.
> A test that has not been characterized and qualified produces unknown results
Known and unknown are not binary states. You can have a new test which is more likely to work than not based on all currently available evidence and you have to choose whether to use it during the period when better data is being gathered. The best answer is not automatically no.
That is different than the problem for approvals in general, but still the same. In the normal case you have to choose whether less rigorous use is allowed in early stages which provides more data prior to making the decision of whether to even do the more expensive more rigorous trials, thereby making the rigorous trials that are actually performed more likely to have positive results and increasing the number of effective treatments on the market.
Because everyone else I've read in the medical industry made the argument they (and the CDC) were a large cause of why our testing was behind other countries. (this includes a previous director of the FDA)
I don’t understand how someone could argue that establishing that a drug works and will not harm you is “red tape”. I won’t comment on COVID testing because the facts are not established and the depths of incompetence and criminality associated with the topic are yet to be discovered.
How is "informed consent" (to the extent that that is even possible) not possible outside the context of clinical trials?
And "medical monitoring" doesn't mean it won't kill you, it only means if you die they write it down.
> I don’t understand how someone could argue that establishing that a drug works and will not harm you is “red tape”.
It prevents you from using a drug that could work based on no evidence that it doesn't. That is a much more aggressively authoritarian stance than prohibiting something which is actually known to not work or be harmful.
It also prevents you from using things that are known to work, and are used in other countries, but have no one to pay for the US clinical trials because no one holds a patent on it, e.g. because it's a part of a plant.
I think perhaps you've never run a clinical trial. What's involved is far beyond what you are describing.
Monitoring can't save you when the damage is already done by the time the evidence of damage is discovered.
We live in a world where Alex Jones exists and people will use religion and hocus pocus vitamins/supplements to peddle fake medicine.
Those charlatans live in a regulatory loophole that allows dietary supplements to be marketed in fraudulent ways. So it’s pretty obvious to figure out what would go wrong with medicine if you care to look.
Most of the issues you are worried about are more about the business model of medicine.
At least 1. Contradicts most what I have read, which is that our speed of innovation/knowledge acquisition is accelerating
It would be easy to say it all should be standardized (and it should, for the most part), but workflow considerations keep standards from being adopted. What a data standard for oncology data would look like is very different from what one developed for ophthalmology. Not to mention UX - if you’re a doctor seeing 20 patients a day, an extra 15 minutes of data entry per patient could bring that number down to 15 patients or less. UX and data standards really does have a direct link to patient outcomes in healthcare!
I'm not surprised that patient compliance is a problem, but it still seems like an expensive burden to find patients physically near the clinical trials so they can come in to take the medication. It just seems like a logistical nightmare that software could help with.
Could you build an app that requires patients to record themselves taking their medication under whatever conditions they're supposed to? You might have to manually audit all of the recordings, but the number of people that still fail to comply might be offset but how many more patients you can include in the study. There might be a lot of other things you could verify remotely using wearable tech, e.g. heart rate now that Apple's EKG monitoring is FDA approved.
Great read, I'd like to see more "Chesterton's Fence" arguments for other highly regulated industries.
Yes. I work for a company (Signant Health) which does clinical trial platform as a service (among other things) and our platform has this feature that the trials can have steps which require the users to video certain things.
And we are hiring https://signanthealth.com/careers/ :)
But compliance is certainly hard to assess and so important. Your app thought reminds me of "Video Directly Observed Therapy" which is commonly used for things like tuberculosis medications (due to painful combo of lengthy duration of treatment, disproportionately affecting less well-off / well-educated people who are more challenged in compliance, and public health risk if inadequately treated).
Several companies have also worked on pills with an component that detects when it's been ingested and beams a signal to a (external) device, thus "confirming" ingestion.
Although of course, in a way, compliance is part and parcel of your drug's overall effectiveness. Not to dive into efficacy vs effectiveness and the larger considerations... but what if your drug "works" but nobody can be compliant? Of the top of my head, a few antibiotics are like, dosed 6 times a day, so I'd rather prescribe most patients a different one that's only taken twice a day, even if the 6 times a day antibiotic is technically "better" for their infection.
This itself had to be approved and has a number of ways it can be prone to errors and blow a trial by misreporting if it were used for one.
Generally, especially under the current circumstances for at-risk patients, there might be a push to do more electronic reporting for clinical trials. However, the requirement to have patients near a trial center isn't that bad, you don't have to be right next to it. But when patients drop out of trials it is very expensive to the company so reducing travel problems is important in addition to maintaining consistency.
Do you have any more information about this please?
Wouldn't that be easily fooled by just pressing the pill against your abdomen for a while, and then throwing it away? Or sticking it with a piece of duct tape?
Seems like this solutions kind of assumes that all pshychiatric patients are stupid.
It would be difficult to game this but it is still possible. The app itself could be broken for example.
I think the FDA approval for this device assumes the patient actually takes the medicine so no guarantees are made if the patient doesn't.
To tie this to my point, adding in electronic reporting and measuring opens up a number of possible ways of getting false data during a trial so I think it would be difficult to manage that. For example, in oncology trials, monitoring for adverse events and cancer progression might involve blood draws for analysis and maybe a monthly progression scan. These types of things are much easier controlled at a central location. Although not all trials are the same and electronic monitoring and reporting seems to have a lot of potential for drug usage and trials.
We've had such a case a short time ago (months?) in the news. Does anyone remember what that was about or even has a link?
It actually can happen quite frequently. Trial protocols can pre-specify early stopping criteria: if we see drug A is better than drug B by a factor of X at time T, then we can declare superiority with given statistical confidence.
I was considered for (but ultimately did not qualify IIRC) a trial of a new Hodgkins treatment. If I had qualified and opted in all that would have meant is basically they toss a coin, heads I get the new treatment, tails I get the existing "Gold standard".
As well as not using a placebo because it's a death sentence, they also can't truly blind the trial because the new regime is different drugs, with different side effects and a different calendar. The staff must know which they are giving you and even if they didn't just tell you it's not hard to look up online what the Gold Standard is and tell if you aren't getting it.
So all they have left it's a two arm trial in which they've tried to match the arms up, and then compare the results.
In my case the eventual trial results show the new regime was slightly worse, not better as hoped. Slightly more (statistically insignificant but not good news) deaths and worse side effects. Unfortunate, but there was only one way to find out.
EDIT: adding details:
You have to mobilize large interdisciplinary teams to come up with products that are inherently very hard to invent, and which, after all that, can easily fail a trail and never see light of day, or can then still harm or kill people even after approved, all while operating in an understandbly extremely regulated and complicated environment.
One example:
One of the many piles of documents that you need to submit to a national health authority _just so they consider your product_ (before you can release to a single "user"!), is thorough documentation of how you _plan_ to manufacture the product, if it were approved.
Just these text files can weigh _gigabytes_. Think how much effort it would take just to write something coherent that is so long! And of course, you have to get other teams to run a bunch of experiments to inform what to write.
And this is even before you are actually producing the product for release.
Oh, and maybe your product won't actually be approved, so all that goes to waste.
And that's just the beginning.
Because all of that excludes _any_ mention of the real deal, which is whether the product actually works or is safe. All of that requires its own experiments and documentation.
And all that assumes that decades of prior research actually led to some useful prototype where you can start.
And all that excludes any mention of ethical considerations like who pays for the research, what is a just price for a cancer drug - heck, whether you should even be trying to make a profit off of lifesaving drugs.
If getting to a point like selling your app called Whats App for $18 billion (or whatever it was) would require a walk down the street, then getting a drug to market is like leaving the earth, flying past the furthest planets, and then trying to reach another galaxy.
So many really bad drugs that are popularized and thrown with black-box warnings and such after the fact.
I think the integrity of that system has been subverted.
This is why drugs are so expensive in the West.
Because if you really want to prove that it works and is safe, you have to give it to a lot of people, all of whom have different health conditions. To establish that, you would ideally give it to everyone with the disease of interest. But you can't test a drug on the entire world, so.... you only find so much in a clinical trial.
(I also agree that having large amounts of cash and runway is necessary for a successful trial).
Right now, the cost of R&D in the drug market is such that even companies worth billions recede before the task. The classic argument regarding that is too much vs. too little regulation. Unfortunately, no one can decide the truly necessary amount of consumer protection, so you end up with a balance between no innovation vs. Thalidomide.
And it’s simultaneously easy to convince yourself you got it right when you didn’t.
In addition, I suspect that you have never debugged a delicate physics experiment.
A single experiment could fill a dozen notebooks with records of what I tried and used. There were records of serial numbers, calibration sequences, colors of wires, etc. I could go on and on.
Why?
Because when something didn't work, I needed to have "ground truth" as to what was working so I could focus on the (hopefully contained) surface of uncertainty. And this was for a physics experiment where the outcome is KNOWN.
If you don't actually know the outcome, everything gets worse as you don't have a definite signal between "Hypothesis is wrong" vs "Experimental setup is broken". Now, multiply this by thousands of steps for a drug. Brrrrrrrrrr.
To top it off, the physical world isn't like software, you often can't go backwards. In the physical world, if you didn't record everything on the forward pass, you may have to go back to start. :(
A: If you allow a drug that ends up causing more harm than benefit
B: If you ban a drug that would have caused more benefit than harm.
So making these regulations is about finding the balance between A and B.
The problem is that a type A error causes a huge scandal. Those who took the drug and died are known by name and there is terrible publicity.
But a type B error only causes people who would have died before to still die. Nothing new has happened, and there is no outcry.
So the pressure on FDA to err on the side of never having any type A deaths, while allowing possibly millions of type B deaths is enormous.
So I tried to move on to the substance.
It's of course true that science can be very hard. I'm... not really connecting the dots?
No, you're not. And that's why you're missing the point that the details ARE the substance.
The voluminous FDA regulations surrounding drug approval are about tracing EVERYTHING about that drug so that when (not if) something goes wrong, you can figure out what went wrong.
Theoretical Example: 5 people just went blind from eyedrops of a drug that has never exhibited that problem before.
Should the drug be pulled from shelves?
If we work backward through the pipeline records and find that the 5 people all share the same batch of eyedrops, yes. If they don't share the same batch, no, but then we probably need to call the CDC as we have a new disease.
But, what's a batch in common? There is the crate that got delivered to the pharmacy--did the drug get too hot and melt some of the conatiner into it? There is the batch that got delivered to the distributor--did a rat or something urinate on the box? There is the batch that got delivered to the sterilizing company (probably gamma radiation for eyedrops)--did they not give it enough dose or did they have a metal obstruction from multiple loads in the facility.
I can keep going further and further back. Was the water in the process supposed to be deionized or sterilized in some way? Did somebody mix up two similar compounds on the line? Did you clean your reactor with the wrong compound? Some drugs are made by cells and your reactor can "drift" over time and you may have to purge it and start over when the drift gets too far.
The regulations demand that you record all of that up front before you can even BEGIN the process of getting a drug approved for use.
(The cleaning the reactor one is not theoretical: https://blogs.sciencemag.org/pipeline/archives/2007/07/09/no... )
In addition, I would like to add that the FDA are actually really helpful once you demonstrate that you are not a clown. However, they get a LOT of clowns on a daily basis and basically have to assume that you are too until you demonstrate otherwise.
> I can keep going further and further back. Was the water in the process supposed to be deionized or sterilized in some way? Did somebody mix up two similar compounds on the line? Did you clean your reactor with the wrong compound? Some drugs are made by cells and your reactor can "drift" over time and you may have to purge it and start over when the drift gets too far.
These days we could track every single step in this product supply chain far more effectively with a crypto-based, blockchain approach. It would be a lot more efficient.
Tracking isn't the hard part.
Documenting "what to track and what it should be" UP FRONT is the really hard part.
The goal is to set things up so that when your system scans a QR code for that barrel of "reactor cleaning fluid" and it scans as methanol instead of ethanol the system starts screaming at you.
You can do all the steps by hand with a paper register. Nobody cares what your recording substrate is. However, they do care that it gets recorded in such a way that it can be verified.
And I would say "Someone has a problem and thinks "Oh, I'll use blockchain." Now he has 2 problems."
I've just seen it recently with a diary company where a friend was working. Company was build from the ground up by an elderly couple, they sold to an investment conglomerate when they wanted to retire. These guys brought in a new CEO, CFO and upper management team all from the tech startup world. So these guys come say, we must change this, that make everything more agile, streamline... When the production manager kwho had been there for >15years) and my friend former logistics and deputy GM said this doesn't work like this, we need to take things more slowly, do one change at the time, they both were let go, who likes a naysayer. Well 1 year later the company has lost half its value, the investors fired the management team and brought in some people with dairy experience, it's still 50/50 if the company will survive.
The point of this is, the people doing this stuff aren't stupid, there are often good reasons why things are done in a certain way, so stop and listen before being a smug knowitall.
You can solve precisely none of the problems with a blockchain approach.
A blockchain might be used to implement a tracing system, but the blockchain is "just" a storage and query layer, not the interesting and hard part of the problem that needs to be solved.
Clinical trials have 3 phases and it all have to be explained via statistic (design of experiment) and reproducible. They have standards for everything to enter the US drug market.
Lots of biostatistics stuff too.
Even when the drug enter the market, pharma companies stop testing the drugs. It is up to the FDA to keep track of side effects and toxicity that were not caught through the phases. (example: ambien sleeping drug)
The people at FDA NCTR are wonderful, highly recommend interning there for a summer.
They have some ML stuff. I've seen a project where they attempt to figure out what bug is in the food. One of the bug was midly toxic and they wanted a image classifier to figure it out via picture of a bug part. You can only have a certain ratio of bug parts in food.
Often I think it's just bureaucracy, and incompetence. Even if drug/therapy is massively effective, I'm sure the positive results would be lost in the bureaucracy.