Hacker News Reader

Comment by ekianjo | Hacker News Reader

> "Phase 3 double-blind randomized clinical trials are needed to provide definitive data."

Let me LOL at this statement. If you think Phase 3 studies can't be hacked in one way or another, well please meet the Pharma industry with its hordes of 1) cherry picking patients for phase 3 recruitment, 2) cherry picking patients who should be removed from the analysis as "dropouts", 3) p-hacking and multiple other practices well known under the sun.

Believing that randomized Phase 3s remove all bias is incredibly naive.

View on HN

Based on my experience (I designed pharmaceutical clinical trials in an earlier life) this is much harder todo than it sounds. You have to start early (Phase 1) and what you end up with goes on your label (basically the package insert). You don’t always end up where you want to.

The real shenanigans appear once your drug is approved.

> this is much harder todo than it sounds.

Cherry picking patients is not hard, since this is entirely up to the trial's recruitment criteria. This is also what makes comparing drugs virtually impossible since no 2 drugs in the same market apply the same recruitment criteria. This is how manufacturers invite the discussion about "superiority" while competitive drugs are almost never compared directly in the same trials.

I wonder if there are cryptography based secure protocols that can ensure that the studies have been carried out the proper way. Given your experience would love to hear your thoughts.

My second question to you is to understand what apart from deep pockets prevent a trial to be repeated until an outcome favorable to the drug company is obtained. I understand pharma companies are encouraged to make all their trials public, but to me that seems like a generic guidance with no teeth.

> I wonder if there are cryptography based secure protocols that can ensure that the studies have been carried out the proper way. Given your experience would love to hear your thoughts.

Cryptographic methods would be overkill and yet couldn't be applied to the places where such fraud might occur. When you submit trial results the documentation is immense -- you submit pretty much everything. There is a long, highly documented chain of custody and a lot of people (and independent entities) involved so it would be very difficult to pull off a conspiracy. That's not where the problems lie.

For example if the agency thinks something is irregular they can go back and look at the underling data and the instructions on how those data were collected.

The amount of data supplied is so large that before the days of electronic filing every drug company had a group whose job was simply to manage the logistics of organizing the trucks needed to transport a single submission! That sounds therefore like there would be a needle-in-a-haystack problem, and of course there always is, but you'd be surprised by the tiny details you get asked about as someone sees something they don't understand or are suspicious about and just follow the trail backwards.

> My second question to you is to understand what apart from deep pockets prevent a trial to be repeated until an outcome favorable to the drug company is obtained. I understand pharma companies are encouraged to make all their trials public, but to me that seems like a generic guidance with no teeth.

So this is the place to look: at a high level. However what you ask is hard. You can't legally give humans unapproved drugs (the agency doesn't give you permission to do so; they say "you proposed to use X on people who meet criteria C, following a specify procedure P, in order to answer questions Q, R and S. As long as you do precisely as you asked, only on the population C and on no more people than discussed, we will not prosecute you for using an unlicensed drug". And you submit an enormous amount of data you have already collected that explains why it's worth giving it a try and why it isn't a risk to human life.

So you do specific pre-clinical (animal) work just to get it into humans at all (phase 1), then use the safety data you collected to justify some kind of limited study of efficacy (Phase 2) and based on that you do a large, representative (of the people in the US who might need this drug) population of subjects.

If you tried to repeat a trial the agency would say "but you already did this and didn't like the results, so no". And in fast why would you? You'd be desperately trying to figure out why it failed and try to do something different.

Thanks a bunch for your response.

Regarding cryptographic methods, I was indeed clutching at straws. It seems to me that current methods can convey that there are 'no obvious indicators of fraud', it would be nice if pharma had recourse to methods by which they could 'prove' that fraud couldn't have happened. I suppose it would be quite impossible to design such a protocol, 'best effort' is probably all one can do. I think mandatory pre-registration itself fixes many of the potential problems.

On a different note: compilers and medical trials … I don't see those going together a lot.

> On a different note: compilers and medical trials … I don't see those going together a lot.

Or power generation or... I have enjoyed working in different domains requiring completely different technical skills. I have learned a lot.

It's one thing to learn chemistry and lab techniques and protocol. It's another to present before the FDA who doesn't have any interest in what my previous career might have been.

Some folks, perhaps many folks like 'more wood behind fewer arrows' culture. I lean the other way, so I take interest in stories where people have moved from things to other things. For understandable reasons CS seems to have a lot of them.

The Google reference is a little ironic, because at one point in time, say around 2004, it was a place where anything that's intellectually challenging and gratifying was fair game.

I would assume Bell Labs would have been something like that.

> I would assume Bell Labs would have been something like that.

Talking to people who worked there it seems it was much less so than it would appear. PARC was more like that.

Perhaps phase 3s aren't perfectly free of bias, but they are essentially impossible to effectively manipulate by the drugmaker.

Even if we assume all the physicians involved in the trial are willing to be bribed, the pharma company doesn't know which ones are handing out placebos.

> Perhaps phase 3s aren't perfectly free of bias, but they are essentially impossible to effectively manipulate by the drugmaker.

Not completely accurate.

Studies with negative outcomes are not required to be published by the drug maker. You can virtually keep trying the same drug and "get lucky" with a single trial, and just register that one with the authorities since the standards for replication are still very low.

No because you have to preregister all of them.

Preregistering does not mean anything about sharing actual results. the large majority of negative outcomes are NEVER published. Please don't spread misinformation.

Pretty sure FDAAA has mandated reporting results for a while now (although some sponsors choose to break the law). Do you have an example of your scenario of an approved drug with many phase 3 trials pre-registered but only one with reported results?

> pharma company doesn't know which ones are handing out placebos..

Not sure I get you. If all the physicians are bribed, then why would the company need to know anything? Everyone can cook up some data and go along with the result that the company needs, right?

Also, about this

>If all the physicians are bribed

I think you should assume that any physician that ends up in a trial is there because they are known to be complaisant. Same as politicians and authority figures are often planted by powerful people with vested interests.

By phase 3 they should have to preregister the study and identify the outcomes they will measure. There isn't much room for p-hacking at this point.

There will certainly always be bias, but it's still a gold standard for good reasons.

>a gold standard for good reasons.

What does this mean exactly? Is it the best possible? Or the best available?

If it is the latter, then it does not mean much. If it is the former, then your statement is not true.

I mean "good reasons" as in objectively good reasons. It's not a comparison. Preregistration is one of those reasons.

> By phase 3 they should have to preregister the study and identify the outcomes they will measure. There isn't much room for p-hacking at this point.

Not all Phase 3 with negative outcomes are published. It's actually the opposite. Most of them are not. So preregistration achieves nothing in terms of transparency.

And p-hacking still happens far more often than you think - I have personally seen trials where they identified "sub-groups" of patients in order to prove that there were patients for which said drugs worked. That's a thing, no matter what you decided to set in the first place as objective.

And then there's stuff like Biogen's alzheimer drug for which no efficacy was proven but still received FDA commercialization approval. Everything is working just peachy, right?

Necessary but not sufficient. Those are issues that can plague all studies, but at least some sources of bias and error can be mitigated against with randomisation and blinding.

Reply on news.ycombinator.com