Many FDA-approved AI medical devices are not trained on real patient data
medicalxpress.com
medicalxpress.com
The lack of "published" clinical validation studies implies neither that the AI developer performed no clinical validation nor that the FDA hasn't seen it. So, it is not clear if the problem is with the lack of clinical validation or the lack of reporting. For some reason the title exaggerates yet further (half of FDA-approved AI not "trained" on real patient data).
if the developer went through the trouble and expense of performing clinical validation you can be sure that they would publish the results __unless__ the results reflected negatively on them
Sometimes they do publish the validation data, especially in pharma etc. Most times they don't, why would you? It would only help your competitors.
It's tricky. On the one hand, it's obviously not appropriate to be flippant about patient privacy. On the other, it's clearly that advancements in human health are being hindered by our current approach to (dis)allowing researchers access to data.
It's hard to overstress enough how impossible it is to do even basic research across institutional health datasets, even you're a giant organization with a compliance team. It's soul-draining and frankly the reason a lot of smart people jump ship and work in finance or crypto or whatever, where you can accomplish something even if it's goofy.
- Insurers won't insure you
- Abortion activists will hunt you down
- Religious fanatics will shame you
- Credit rating
Not to downplay those (very real) risks in the slightest, but they are all US-centric problems.
Not sure what over-arching point to make, but it's certainly a set of US-centric problems.
And there's the problem. In theory, this is possible. But in reality, there is no such thing.
You also assume that your data will be correct. If data integrity was so easy and common, people wouldn't be encouraged to repeatedly check their credit reports for mistakes.
One bit flip and you go from "perfectly healthy" to "about to die" and suddenly you can't get life insurance, your credit score tanks and you can't get a job.
There's a lot of money being spent on deanonymizing data, and I would never count on it ever being able to remain anonymous with that much incentive.
Since these models are black boxes, it’s easy to hide biases within them
Same negative attached to pre-crime in policing because people that wear the same clothes, drive the same car, listen to the same music, and other sames have committed crimes, we think you will too. someday
https://www.hhs.gov/hipaa/for-professionals/special-topics/g...
https://www.hhs.gov/healthcare/about-the-aca/pre-existing-co...
[0] https://assets.publishing.service.gov.uk/media/624ea0ade90e0...
They got acquired by a major pharmaceutical company, but I don't know of any major discoveries made from their data. Because even with real data this is a hard problem.
The FDA keeps a database of all adverse events reported by manufacturers or healthcare systems [1], so we can check in a few years if these AI medical devices are causing an uptick of complaints.
[1] https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfmaude/s...
"Oh, your great great-great-grandfather had <disease>, so now you're classified as a high-risk individual and you have to pay a higher monthly fee for insurance"
Only they won't tell you why you have to pay more, just that you do.
2. If health insurance companies wanted to break the law, they would simply violate the existing prohibition on discriminating on preexisting conditions, which gives them vastly more actionability than some tenuous, diluted link to a relative.
This is all frankly nonsensical because insurance companies charge people within tightly-regimented tiers, there's no wiggle room for mystery +30% fee increases.
People don't want their info used for this without their permission.
Freedom
Knowing if you have an expensive condition could be invaluable to say, an insurance company.
Knowing something else may give people the opportunity for blackmail.
There are lots of reasons one's medical history is valuable. There is more that is valuable than just money.
Why does this has to be so hard? I mean what if the regulators went the opposite approach: make ALL the data publicly available by anonymizing it.
What privacy concerns would anyone have if the whole data is completely anonymized? Wouldn't it create far more benefit by accelerating innovation in healthcare? I strongly believe the privacy concerns (which shouldn't exist after stripping out any PII) are outweighed by an order of magnitude by the upsides, which are literally saving millions of lives in the longer term.
But again, regulation, as always, cripple innovation, this time slowing down developments that would literally save lives. Great.
There are various long articles on this and it remains the case that patients have a right to confidentiality.
it's frustrating that the article uses the term "approve" which is specific to PMA devices (class III), whereas class II devices are "cleared" by the FDA.
- a medical device executive
They told me that FDA has nothing to do with actual product testing, e.g., human trials. What the "approval" guarantees is proper facilities and other second-order criteria.
I'm writing this comment in the hopes of learning that I have misunderstood something crucial here. Because if it is indeed the case that the companies themselves are the only ones vouching for drug safety, that old story about capitalism and incentives implies we're seriously fucked.
additionally, product testing != human trials. they are two very different things.
final product testing is called "design verification and validation" (aka dV+V) where a statistically significant number of devices are tested against several different categories.
human trials can run the gauntlet from a small IDE studies to phase I through IV.
the drug approval process is MUCH different than the medical device process.