Hacker News Reader

Comment by DarkmSparks | Hacker News Reader

Massive, major flaw in their argument. "without seeing the raw data" is not true. At least in the UK.

All raw study data, published and un published must be provided before a medicine will be reimbursed. It is reviewed by teams who do their utmost to prove the med is worthless.

Same for the FDA afaik. All the data is generally "commercial in confidence" so any old twitteraty can not download and misinterpret it.

All trials must be registered on Clinicaltrials.gov or they cannot be used as evidence.

Publication bias is a well know issue, resolved for more than a decade. No idea how this article made it into the BMJ, smells like click bait.

View on HN

"All the data is generally "commercial in confidence" so any old twitteraty can not download and misinterpret it."

This is totally the wrong attitude. Everyone should be able to look at the data. The established authorities are absolutely capable of totally misinterpreting data. The only hope is to let other people look at it.

In theory I agree with you.

In practice, Ive seen too many bad faith actors intentionally drawing false conclusions and then linking to data they say agrees with them that clearly doesnt. Twitter, wikipedia and facebook are terrible for this behaviour, many of the so called fact checkers just as bad.

Healthcare trial data analysis is a highly specialised skill, (last one I did took about 3 months of meetings just to understand the dataset, about 6000 columns of abbreviations of complex medical terms) those with the qualifications to analyse it are typically not refused access if they actually want it, something like GPRD is generally more valuable and available to everyone.

In Sweden everyones entire medical history is open access for research.

There isnt any shortage of data, there is an acute shortage of analysts and finance to cover the costs involved.

I have seen many studies where they say that qualified investigators can have access to the data if they sign lots of forms, agree to collaborate with (not just acknowledge) the original authors, and wait six months for this all to be approved. I expect that I would be approved if I did all this, but it's not worth it. In some cases I suspect the study is flawed, but debunking it months later would be a lot of work for little return. In other cases, the study looks OK, but without ready access to the data to confirm this (and perhaps get more information with a more sophisticated analysis), it's really pretty much worthless.

Yes, worthless. As in no one should put any weight on such unconfirmable "evidence". It's just wasted effort by the investigators, who piddled away their time (and their funding agency's money) because they're not willing to release the data.

These are not studies where one would need 3 months to just understand what the variable names mean - perhaps the data was extracted from such a huge database, and perhaps there are issues with how that was done, but there are also quite possibly issues with what they did afterwards, which can be examined separately.

Comments by unqualified people on twitter are a fact of life. You can't get rid of them without also eliminating comments that reveal mistakes (even fraud) in what superficially seems like authoritative research. Your judgement that such uninformed comments are intentional lies is probably untrue in most cases.

>Yes, worthless. As in no one should put any weight on such unconfirmable "evidence". It's just wasted effort by the investigators, who piddled away their time (and their funding agency's money) because they're not willing to release the data.

And UK regulators definately do not. If they dont have sufficient data to be confident in the decision they make collecting more a requirement of reimbursement.

Here is an example of the guidelines

https://www.nice.org.uk/process/pmg20/chapter/reviewing-rese...

Taking only a single trial is worthless. Or rather about as useful as taking a murder suspects mothers testimony. Evidence based medicine is simply not broken by not handing out complex, expensive and commercially valuable data to everyone who would like a copy.

Any more than the court system is not broken by keeping certain evidence from a jury.

>Your judgement that such uninformed comments are intentional lies is probably untrue in most cases.

Oh they definately are. Just look at the recent data on covid vaccines published by public health wales to see how bad that situation currently is.

> Publication bias is a well know issue, resolved for more than a decade.

How is that resolved? This is not what I see in my meta-analyses.

requiring registering with clinicaltrials.gov and regulators doing their own analysis on all the data.

Managed entry agreements and conditional reimbursement (dont pay for patients that dont do better than the average of current practice)

Requirements for "real world data" collection and submission post approval.

A whole host of other mechanisms I cant remember off the top of my head.

Also

"evidence based medicine" doesnt mean the evidence is infallible (e.g. rare disease meds have a huge issue with collecting any meaningful data) - it just means actually using evidence rather than just taking the industries and acedemias word for it that it works like the old days.

From personal experience, I think you are massively overestimating the reach and effectiveness of regulations.

I've worked with nice. several of my students still do. We've kicked out of the NHS quite a few meds the industry and academics were wrong about, based on the evidence, not what the industry/academics said before they were shown to be not correct.

The whole posit of this article is wrong. "acedemics said so" is not evidence based medicine, it is one small facit of it, and the author matters, no one trusts "sponsored by" papers, NIHR is constantly publishing counter evidence. Some of that even ends up in court (e.g. Avastin)

https://www.reuters.com/article/us-novartis-bayer-britain-id...

This I totally agree with. I've not much experience with the UK system, but I disagree that publication bias is solved, at least regarding the scientific literature at large.

Well. I guess you could probably pin some blame on publication bias for the massive misinvestment that went into the recent dementia meds that all failed at phase 3. Something made them get the science badly wrong.

I was talking specifically in the context of evidence based medicine for regulatory decision making, which is the process that replaced blue envelopes full of money or throwing acedemic papers at an important politician whose relative had the disease that came before it.

Do these issues not still apply? Have they been fixed?

https://en.m.wikipedia.org/wiki/AllTrials

They are "fixed" by extending the evidence base for decision making well beyond raw clinical trial data and publications on them (none of which ever translates into real world practice outcomes anyway).

In important instances (such as expensive meds going to large numbers of people) the payer will commission their own research e.g. through NIHR https://www.nihr.ac.uk/

There is a whole field of "value of information" now to estimate how incorrect information would affect a decision and whether it warrants research investment (which covers publication bias - e.g. how many unpublished studies showing no or negative effects would there need to be to change the decision)

Reply on news.ycombinator.com