Yes, Clinical Trials Work | Hacker News Reader

Yes, Clinical Trials Work

simplystatistics.org

42 pointsjtleek21 comments

A great book RTCs is Ben Goldacre's Bad Science (http://www.badscience.net/)

He points out that some of the first clinical trials recorded were in the bible. I couldn't find his quote, but I found an NEJM piece that references the same thing, although I think its in a nutrition debate.

"The first published report of a clinical trial has biblical origins. In the Book of Daniel,5 reference is made to the unwillingness of the Israelite Daniel to accept the diet offered by the Babylonian king Nebuchadnezzar. The king's official had put a steward in charge of Daniel and his three friends (Shadrach, Meshach, and Abednego):"

"Daniel said to the steward . . . “Test your servants for ten days; let us be given vegetables to eat and water to drink. Then let our appearance and the appearance of the youths who eat the king's rich food be observed by you, and according to what you see deal with your servants.” So he hearkened to them in this matter, and tested them for ten days. At the end of ten days it was seen that they were better in appearance and fatter in flesh than all the youths who ate the king's rich food. So the steward took away their rich food and the wine they were to drink and gave them vegetables."

(http://www.nejm.org/doi/full/10.1056/NEJM200301023480120)

Maybe I'm mistaken, but the historical bias fat=healthy seems to be visible there. It's doubtful that by eating only vegetables they could be 'fatter' than those who drank and ate rich foods. It's obvious the writers were trying to make a point, but I'm not persuaded by the accuracy of the findings.

Yeah, I don't really care one way or the other about the context. I more care about the fact that there was a clinical trial thousands of years ago before concepts of statistics... I like that it can all be boiled down to: split people into two groups at random. Do something to one group, do nothing to the other, measure the results.

I think the blog post missed the most important point of the NYT piece: Too often, he says, trials are against “a straw-man comparator” like a placebo rather than a competing drug. So the studies don’t really help us understand which treatments for a disease work best.

The standard is usually a placebo. This is done when you are not even sure the drug will have an effect.

The standard should be a competing drugs, otherwise you have no measure of whether the drug under trial is better than current drugs.

The financial incentives are not present for any of the parties (including the regulators) to use the higher bar.

Is it not possible to take the competing drug's trial against a placebo, the new drug's trial against a placebo, then compute the new drug's effectiveness compared to the competing drug? Is it just that it's too inaccurate to go through the indirection like that?

I wonder if this will have unintended economic consequences. Clinical trials are astonishingly expensive - if we required all players to clinically test against competitors (or just the current leading competitor), it seems this would create a perverse incentive to be first to market. Not only would you rake in the early profits, but you'll also significant raise the bar for anyone trying to unseat you.

I'm not convinced this will actually be the case, but it's a possible outcome worth thinking about.

One thing I'm wondering: if all clinical studies are quantified via efficacy vs. a fixed placebo, shouldn't that make the results comparable? If Drug A is 200% more effective than a placebo, and Drug B is 300% more effective, does that not suggest that Drug B > Drug A?

> One thing I'm wondering: if all clinical studies are quantified via efficacy vs. a fixed placebo, shouldn't that make the results comparable? If Drug A is 200% more effective than a placebo, and Drug B is 300% more effective, does that not suggest that Drug B > Drug A?

I'm not sure that works. How do you account for different test conditions?

Exactly -- the tests may have been done on different populations. In an extreme case, imagine both treatments are randomized trials for an otherwise terminal illness, but Drug A was tested on the elderly and Drug B was tested on youths. When cured, those on Drug B will live longer, if only because they are young.

Yes, it is possible to compare two placebo controlled trials and get a rough estimate of the relative efficacy of the two drugs. Many of the prescribing decisions that physicians make are based on these indirect comparisons.

However, it's only a rough estimate. Even if the patient populations are nearly identical, you can often see different outcomes.

To give you an example: Crohn's disease is an autoimmune disease of the large (and sometimes small) intestine. If you look at the clinical trials for the biologics used to treat the condition, you'll see remarkably varied outcomes, even in the placebo arm.

In other words, if drug A show 50% vs. 20% efficacy vs. placebo and drug B showed 70% vs. 40% efficacy vs. placebo, which is the more efficacious drug?

> "drug A show 50% vs. 20% efficacy vs. placebo and drug B showed 70% vs. 40% efficacy vs. placebo"

Which raises a further (or more basic) confounding factor: the placebo effect isn't fixed.

It does vary between trials and even appears to be steadily increasing in potency over time. [1]

So first-to-market drugs have an added advantage when naively considering "improvement vs placebo" -- as their test were run years ago, when the placebo effect itself was a weaker opponent.

[1] http://www.wired.com/medtech/drugs/magazine/17-09/ff_placebo...

It may be "too often" that new treatments are compared to placebos, but it's not the prevailing practice, at least for serious indications like cancer. It would clearly be unethical to deny people proven treatments would extend their lives. To quote cancer.gov (http://www.cancer.gov/cancertopics/factsheet/clinicaltrials/...):

  10. Are placebos used in cancer treatment clinical trials?

  The use of placebos as comparison or “control”
  interventions in cancer treatment trials is rare. If a
  placebo is used by itself, it is because no standard
  treatment exists. In this case, a trial would compare the
  effects of a new treatment with the effects of a placebo.
  More often, however, placebos are given along with a
  standard treatment. For example, a trial might compare the
  effects of a standard treatment plus a new treatment with
  the effects of the same standard treatment plus a placebo.

I really would like to see this claim from the article verified. Most clinical trials I'm aware of are not truly against placebo, especially not cancer trials. Even the Avastin trial mentioned in the article is not truly against placebo.

The two arms were: (a) standard treatment + placebo (b) standard treatment + Avastin

So, while it is true that placebo was compared to Avastin, it could be that Avastin acts identically to current standard of care. That is, Avastin could show no impact in this trial and show impact in "just Avastin" vs. "just placebo".

In most progressing fatal diseases, it is unethical not to provide standard of care, so trials are set up such that all patients receive at minimum current standard of care.

Forcing a comparison to existing drugs sounds good, but things that are less effective but have fewer side effects are often far more useful than the most potent option aka Tylonol vs Morphine. The problem is you still need the placebo baseline or saline would pass as the safe but less effective alternative. Now, adding competing drugs to a trial is great information the problem is they are already really expencive so adding even more cost is generally a hard sell.

The thing that freaks me out about placebos is that you can rig them - a sugar pill, for example, isn't necessarily inert in diseases that mess with sugar levels (among other things) - by pitting your drug against a "placebo" that actually makes the disease worse chemically, you can get away with murder when your drug magically turns out to be better than it.

Comparing against other drugs, especially established ones, is a reasonable fix for this, although as others have pointed out this may cause perverse incentives.

I'd suspect that diabetes pill placebos would be something like a gelcap filled with water, not a sugar pill.

This defensive blog post perhaps misses a crucial point that is made by the NYT through a haze of lay-person journalism.

By placing the focus on an 'average' patient, there is a danger of missing drugs which work effectively but only in sub-populations while having costs in other sub-populations (as drugs generally do in practice).

"Some who take Avastin significantly beat the average" is probably indeed worth noting because the naive RCT may be disguising significant effects of clinical importance.

The trouble is that it doesn't help anyone on average if you can't identify whether they're in the relevant subpopulation before you give people them the drug. That's why the FDA doesn't approve such treatments.

Indeed, drug companies slice and dice their data from early clinical trials every which way to identify the best subpopulation to run their next trial on. Their interests are entirely aligned with figuring out some population for whom their drugs work.

I noticed this as well. In an extreme case, we may see very small variance among placebo or control groups and several clusters in the group using an experimental drug. If one or more of those clusters significantly beats the control group and we can accurately classify patients into clusters, that's important information which simple means and medians may obscure.

This reminds me of the importance of looking at data and Anscombe's quartet [0].

[0] http://en.wikipedia.org/wiki/Anscombe%27s_quartet

The article is completely misleading in this regard. Every modern clinical trial looks for these sub-populations. The article even states:

   Sixty percent of the new drugs in the works at
   Genentech/Roche are being developed with a companion
   diagnostic test to identify the patients who are most
   likely to benefit.

That said, it is extremely frequent that clinical trials that follow up on these sub-population observances find them to be statistical anomalies. When you have 100 ways to subdivide your populations, you expect to have some anomalies at a typical 5% confidence level.

The article points to I-SPY 2 as the future of clinical trials, making it sound like bayesian analysis is something novel to solely this clinical trial. As is alluded to in what you call the "defensive blog post", this is now standard practice in clinical trial design at most pharma companies. These companies don't like to throw 9 figures down the drain and they recognize that adaptive trial designs can get them statistical significance at lower cost while determining optimal treatment.

I call it "defensive" because the answer is not simply "yes" and I think it's a poor position to view clinical trials uncritically.

Clinical trials may indeed work, but it's absolutely not clear that they are optimally designed.

Although few people on this thread are experts, there are a plethora of different comments which illustrate the impossibility of perfect clinical trial design - there are always improvements that could be made and subjective decisions that have to be taken.

The point that you make about pharma companies being profit-making companies could equally be phrased as a very strong criticism: that any pharma clinical trial design is likely to be heavily biased by their desire to sell their most profitable ("optimal") products.

What I found most worrisome about the NYT piece was the idea that "the payoff for a successful Phase 3 trial can be so enormous that drug makers will often roll the dice".

Now, IANAStatistician but from what I understand there's an inherent (small) chance of any given trial showing a drug is effective. Wouldn't that mean that since there's such an incentive to test unpromising drugs, it's more likely some would get through by chance and get to the market?

Reply on news.ycombinator.com