A copy of a copy of a copy: FDA medical device clearances | Hacker News Reader

A copy of a copy of a copy: FDA medical device clearances

wcedmisten.fyi

59 pointssurprisetalk59 comments

The site is a little misleading, saying "Several of these devices even led to patient injuries including bleeding, organ puncture, and even cobalt poisoning." The majority of documentation in the 510k process is to mitigate harms to the patient. The catch phrase they use is "safety and efficacy". FDA doesn't really care if your device works as well / better / etc. The market will decide that. FDA cares that you don't injure people more than necessary / more than the predicate device does. (If that sounds strange, consider say a biopsy needle.)

The reason 510k is so popular is that introducing a completley new device is incredibly costly typically requriing a PMA (premarket authorization) requiring clinical trial data. If I'm making a new ultrasound machine I don't need to show that ultrasound works -- that's known. I need to show it works as well as one expectes an ultrasound to work, without danger to the patient. Same as if I'm releasing an updated version, re-trialing that doesn't make sense.

Honestly a lot of the documentation requirements are absurd -- you'd think "why would anyone do something so badly we need to document that we didn't do that"... but sadly most rules exist becuase of corners that were cut in the past...

> The catch phrase they use is "safety and efficacy". FDA doesn't really care if your device works as well / better / etc. The market will decide that. FDA cares that you don't injure people more than necessary / more than the predicate device does.

That's just safety.

> FDA doesn't really care if your device works as well / better / etc.

This is the "efficacy" part of "safety and efficacy".

> This is the "efficacy" part of "safety and efficacy".

Yes. FDA 100% cares if your device works as well. "Works as well as the predicate" is the foundation of the substantial equivalence paradigm that underlies the 510(k) process.

Better is a different story. Legally, they _can't_ care (in a 510(k) anyway), if it works better or not.

Hi, author here! I think my main concern with this process is that the predicate itself might never have gone through clinical trials.

So even if the new device works as well as the predicate, that predicate might be equivalent to a whole chain of devices that have never gone through clinical trials.

This might be fine if the device is low risk, but the 510(k) process has a history of clearing devices that have resulted in patient deaths.

Of course it's a balance between too much red tape and having unsafe devices, I was just personally surprised at where the FDA draws the line today.

Did you thoroughly read through the linked sources?

Because in source number 5, it mentions that some fraction of Class III submissions are cleared through the 510(k) process and not exclusively through the PMA process.

> In 2007, Congress asked the Government Accountability Office (GAO) to review the 510(k) process. The resulting 2009 GAO report described the 510(k) process as less stringent, faster, and less expensive than the PMA process and concluded that 66% of Class III submissions cleared through the 510(k) process in recent years were “implantable, life sustaining, or of significant risk,”

> Because in source number 5, it mentions that some fraction of Class III submissions are cleared through the 510(k) process and not exclusively through the PMA process.

Yes, these are what are known as "pre-amendment" devices, referring to those types of devices that were legally marketed in the US prior to the 1976 amendment to the FD&C Act that gave FDA power to regulate medical devices. FDA was given power to require these devices require PMAs via rulemaking, and has been slowly (far too slowly, in the views of many) closing this loophole.

I'm pretty sure the GAO report is largely referring to Class III devices approved after 1976 in this section... under less restrictive rules of some kind.

I've worked on the hardware and firmware design and implementation for many medical devices, almost all of which were approved via the 510(k) process. I always found it a mixture of amusing and bemusing that we'd have to tell the FDA, effectively, "Look at how similar this thing is to existing stuff!" while at the same time telling the Patent Office, "Look at how new and novel this thing is!"

I've worked on medical devices. As a techie, what surprised me more is not the lack of trials for new devices. It's that for software (and other kinds of components), it's often acceptable to have black box testing to certify a component as functional.

This means basically saying "I pressed a button and the thing did X". That's the test. No need to understand how that works, or provide any more technical documentation or specs, or record system state or variables, or do anything else. Press button, it does X. That's good enough to be certified.

The reason for this is simple enough: sometimes the thing you're using is proprietary and its manufacturer/vendor simply won't give you anything else to certify it. But this black box testing is even used for open source software and components. It's like a short cut you can use to cut through a lot of the testing you could have done. I imagine this will remain standard practice as the "AI" companies push their hallucinating dreck and unexplainable magic into the medical device space.

I work with software medical devices. Another aspect here aside from 510k is that it is required, or effectively required, to comply with something like IEC-62304 (developing software as a medical device), and ISO-13485 (quality management systems for medical devices).

For the scenario you describe the piece that’s missing is risk analysis, a requirement. In preparation to release to market they must evaluate the probability and severity of the button not doing X or doing X incorrectly, and develop mitigations if the risk is unacceptable. What you ask - documentation and specs - exist at some level, but the manufacturer has to define what level is necessary for them. I could see an argument against the manufacturer deciding this for themselves, though it’s likely impractical to do so.

For software medical devices that have hundreds of transitive dependencies it’s not feasible to go at the level you’re describing. Some management of dependencies is necessary but treating as a black box - with quality/test management and risk analysis of the black box - is what the current system defines as a reasonable trade-off. Again I could see arguments for changing this, though for many manufacturers the EU has instituted stricter regulatory in the past ~5 years which has been a bit painful but overall probably a good thing.

Today one of the aspects of medical device development which is under tighter scrutiny is cybersecurity. It’s pretty painful right now. Previously there was not much related to cybersecurity required - obviously not ideal - but the pendulum has swung to the other end of the spectrum making it a significant burden. We’ll see, most of it is adopting new processes which is always painful and slows down progress at first. After the initial hump it should be eased into, and ultimately better for patient care and medical institutions in the long run.

Agree with you here.

Wanted to echo " It's (cybersecurity) pretty painful right now".

The FDA just implemented new requirements. Basically they require penetration testing on all new medical devices. The issue is they don't have the expertise in house to know the technical details, and they haven't defined the tests, etc. Additionally they're isn't yet an ecosystem of partners and service providers yet to provide and compete in providing those penetrstion testing services.

Pragmatically what it means for folks trying to get a device cleared at the current moment:

You need to send your device to the one and only penetration testing house that does this for the FDA now and let them try to physically hack into your medical device. You have to make it impossible and evident if someone tampers with it in any way. This is in addition to all the software security stuff we need.

Imagine of you were making computer monitors. One day you are suddenly required to make it so a technical expert cannot open the monitor up using specialized tools.

From a submission standpoint, as I write this, FDA seemingly cares more about cybersecurity than your medical device actually demonstrating safety and efficacy within intended use. The time that review teams have to review any given device has stayed the same, but fear-driven, heavy-handed cybersecurity regulations (which must be followed) have been added to the mix.

I'm more surprised that medical devices were not previously required to be tamper evident.

They are, and it's generally good practice. The difference is now they spend much more time and effort on the penetration testing.

An example. In the past it would be OK to use security bit screws for this. Yes you can buy the bits online, but it was at least one layer of perceived security.

This doesn't fly anymore.

The real challenge is they implemented these new requirements on devices that were already in the submission process. Also, these things aren't written down anywhere in standards etc. so you know them ahead of time when you design. You have to just wait until the penetration testing and find out.

Ultimately the new rules aren't the challenge, it's the fact that you don't get to know them when you start and finish the design, you find out later.

Isn't the intention that everyone submitting their devices for approval have done in-house penetration tests extensively? Or at least laid out specific claims as to what it can endure and what it cannot?

The third party test seems to be just the last verification stage to reassure the FDA the company is not making unsupportable claims.

Not really because these things aren't defined.

There is no definition or standard to which you would do your in house tests to. It's not like other things where you design it to comply with iso whatever and then you test to that.

Here the standard so to speak is defined by the penetration test itself.

An example in safes. No safe is untraceable. Safes are spec'd by number of minutes to resist a tool attack. Then when a safe company goes to UL or whatever to certify the safe, UL technicians get the best commercially available tools and try there best to break into the safe and time themselves. If it takes them more than the spec, it passes.

Here there is no spec. There is no defined time. There is no standard. It's just up to what you can get the penetration test house to agree to write.

But the company has to submit in writing an application laying out their claims?

I'm not really sure why the lack of such a standard definition prevents people from writing that down and then being willing to back up their words?

I can see a time efficiency argument, cost reduction argument, etc., for standard definitions here, but at the end of the day, they're not necessary.

The companies that offer the most credible products, verified via third party testing, get FDA approval. Everyone else gets weeded out.

>"But the company has to submit in writing an application laying out their claims?"

How so?

Are you unsure about the meaning of submitting a written application?

Or is there some other confusion here?

Please see my reply immediately above.

I'm saying the written submission doesn't contain this, and even if it did there is no one reviewing it that actually knows the technical details enough to provide meaningful oversight.

It's similar to that quote from a Boeing insider that came to light "they (Boeing airplanes) are designed by clowns suprivised by monkees".

Note - these are not my words or opinion, just a quote from another guy

You don’t think they contain… written claims?

I’m not saying they contain foolproof technical specs, but broad claims certainly.

If you genuinely refuse to believe this is possible or is currently done by some fraction of folks, then I guess I’ll leave it at that.

Respectfully, it sounds like the FDA is trying to implement requirements for manufacturers to do what they should've been doing all along. The shitty flipside to my point is that market forces pushed manufacturers to cut costs and externalize the infosec risk onto the patients. The secure products aren't interesting to medical healthcare providers.

I'm, admittedly, a bit salty because I recently looked at a healthcare device that I was prescribed and found evidence that my data is likely being trivially exposed by anyone who wants to look. I can't verify this because it's very likely illegal, and I don't feel comfortable reporting it to the device vendor for fear of being accused of hacking. If there's a way to report it to the FDA, I'd be thrilled -- but I don't know what that looks like.

True. The issue is they aren't specific about it. One penetration testing house may pass a device when another doesn't.

They haven't solved the issue. They've highlighted it and left it up to chaos to solve it.

Did they provide you with Instructions for Use as a lay person? This might contain the legalese on what is or isn't permissible.

The company is required to have a complaint handling process, such that you making them aware of these vulnerabilities would mean they have to at least handle the feedback.

Maybe your findings can be rephrased in a way that don't require you to show how vulnerable their servers are, but that you suspect it's unsafe.

It depends on the risk class according to IEC 62304. When you're developing Class C software you'll need acceptance criteria that goes into how you initialize variables etc.

The next obvious thing to show is the substantial equivalence table. This is where the manufacturers argue that their new device is so similar from the perspective of purpose and technology to the previous (predicate) device that they want to skip trials and a more thorough vetting.

This is where the manufacturers routinely bend the truth by listing the favorable attributes ('both devices are based on Windows PCs') but hide aspects which differ (the materials used for sound insulation for instance in the Philips CPAP scandal).

If the equivalence tables could be shown for each ancestry tree it would become very apparent that the attributes are cherry-picked (I believe).

Do note that the summary pdf page is not the whole submission. 510(k)’s are hundreds of pages. I cannot speak fully for the CPAP foam, but with a predicate you still list every “material” in your device with cut sheets, etc, as well as documented biocompatibility for “airway contact” portions which include the foam. For Philips, it seems the foam degraded into a different, toxic compound which was an oversight.

The author makes it sound like this is a bad thing, its not. Clinical trials are extremely expensive, if every device manufacturer had to do a clinical trial many devices would never see the light of day.

This is a cost vs benefit trade-off. A 510k is not an easy process to get through and is still very expensive and requires an enormous amount of documentation, paper work, and justification.

If a device harms a patient, it become a reportable event, and the FDA will come swooping down and shut you down if you are not properly documenting and mitigating hazards.

The alternative is fewer medical devices saving lives.

Hi, author here! I totally agree there's a cost-benefit tradeoff to be made. I think the 510k process seems appropriate for low-risk devices like latex gloves and bandages.

My concern is mostly around the class II devices cleared through this process, such as the recently recalled Philips CPAP machines, which resulted in 561 deaths. [1]

Philips also knew about complaints without informing the FDA. [2]

[1]: https://www.cbsnews.com/news/fda-sleep-apnea-philips-recall-...

[2]: https://www.propublica.org/article/philips-kept-warnings-abo...

It's ridiculous that a public-facing agency like the FDA does not make every piece of data available in a database that is easy to access and downloadable. You can download all of PubMed in compressed files, ClinicalTrials.gov gives you both API access as well as a publicly available read-only Postgres database, and the National Cancer Institute has excellent publicly available databases. There simply isn't an excuse for requiring someone to spend ~4 months to download this!

Hi, author here! Completely agree, I would love to see the FDA publish an official dataset of predicate device relationships. I'm not sure if they have this data available internally, but if not, they could validate my dataset and republish it, which might be easier than starting from scratch.

You did not have to scrape the pdfs using the website link. They have bulk downloads indexed by year (IIRC). I remember grabbing these because I had to OCR a bunch of summaries to extract data for some NLP I was running. Also, IIRC grabbing the predicates was pretty easy w/ tesseract.

I'd love to take a look, do you have a link? I couldn't find any bulk downloads for this, but that would definitely be useful!

I guess I spoke too soon. I found a resource here: https://open.fda.gov/apis/downloads/

I do use this resource for 510k.fyi to populate the device and recall data, but unfortunately it does not contain the 510(k) summaries, or even links to them. The web scraping was still required to get those documents so that I could run OCR on them.

Since the website is open source, there's even a github issue confirming this: https://github.com/FDA/openfda/issues/200

Yes, it's all about how easy it is to access this data. Many of these older device submissions are scanned PDFs that need to be OCRd.

I wonder if the data could be requested via a public records request?

Live demo (linked at the bottom of the article) https://www.510k.fyi/devices/?id=K121623 (I suggest to add an image of this graph at the bottom ofthe article.)

Great! This is the graph for the Philips CPAP machines which had a big scandal associated with them and which likely got approved through various 510ks.

https://www.nytimes.com/2023/09/07/health/cpap-defect-recall...

One nitpick about the way that 510k.fyi presents this. A 'recall' in FDA lingo doesn't mean that the manufacturer withdrew the device from the market. It just means that they informed customers that there is something wrong with the device which requires any kind of modification by the manufacturer. It is thus a little bit ambiguous to say that a device was recalled if all this might mean that there was some software update which had to be applied.

Hi, author here!

That's a good point, I should probably add a disclaimer about what recalls mean in this context. Unfortunately I couldn't find a good way to determine whether the recall might be a systemic design issue that would affect other similar devices or something that can be remediated for an individual product, like a labeling issue.

I was also thinking of changing the color code based on the recall severity (class I, II, or III)

Hey everyone! Author here, happy to answer any questions about this project!

Do you think it's any better in Europe?

I'm not super familiar with the regulatory landscape in Europe, but I did find this article [1] while I was researching for this project.

I believe the EU has a similar pathway (in the EU Medical Device Regulation) for new devices relying on equivalence to a predicate, but the requirements for demonstrating equivalence are a higher bar. Joint prostheses (like the one causing cobalt poisoning in the US) are not allowed through this process in the EU.

[1]: https://www.sciencedirect.com/science/article/pii/S002013832...

I'm not sure how this would translate into a query, but have you thought about analyzing the longest chain of devices and comparing their PDFs? I feel like that would potentially demonstrate the game-of-telephone effect you're getting at.

That's a good idea! There's actually a paper [1] that does something very similar with the "device description" and "indications for use" sections. They use Word2Vec on these sections and compare similarity.

Fig 2 [2] shows a graph of similarity of all devices to their predicate for each year.

However the discussion sections mentions difficulty with using Word2Vec because of the jargon used to describe these devices, so I'm not totally sure how to interpret the graph.

[1]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8496833/

[2]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8496833/figure/...

One of the jobs I've done in my life was to design and prototype a touchscreen interface for a 1970s style TENS machine with levers and knobs.

They couldn't touch the machine itself, so what we built was a box that had servos and steppers to move the levers and knobs, which was then controlled from an android tablet (using the audio jack as a serial port, at that).

Very Rube Goldberg. It was a fun build.

We've been developing a medical device and going through 510(k) exemption.

Great write-up, and to add to it, 2 things that were missing.

1) 510(k) exemption is where we can point to a predicate device and say "we're so similar to these other devices, that we don't even need to go through the 510(k) process. This is only for low-risk devices.

2) When a predicate device is removed from the market, even if for a safety reason, the devices which received 510(k) approval or exemption are not automatically removed from the market as well!

Having said that, it appears the system is working. The QA and documentation requirements seem to be putting enough administration to provide mostly quality products. It doesn't seem the market is awash in harmful products that don't provide benefits.

With the exception of Philips having been raked over the coals the last few years with their CPAP issues, are there other major issues with medical devices?

Things are handled differently on the pharma side.

Section 505 of CFR Title 21 (which implements the Safe Food and Drug Act) lists cases where drug trials can commence; 505(b)(2) specifically allows the drug equivalent of a predicate device though they don't use that vocabulary.

That allows the safety data of an existing drug to be used, which saves you time getting into Phase 1...and that's it.

I agree that the approval of a tongue depressor shouldn't be complicated, but "not complicated" doesn't automatically imply "anything goes".

I have long believed that a lot of 510(k)s should really be PMAs.

As a tangent, the author says Tesseract based OCR didn’t work very well. Sadly this matches my experience too.

What products (open source or closed) have higher accuracy for vanilla English documents?

Google drive has pretty good OCR

Author here! I was legitimately considering building an iPhone OCR farm for this based on this article about running OCR on memes [1]. Haven't tried it in practice, but the article claims it has better performance than tesseract.

[1]: https://findthatmeme.com/blog/2023/01/08/image-stacks-and-ip...

We’ve built a basic version of this using Notion AI and OCR (https://510k.innolitics.com). Someday we hope to integrate with our more fully featured FDA databases browser (https://fda.innolitics.com).

“The Bleeding Edge” on Netflix introduced me to this idea. It’s kind of terrifying that a huge amount of our medical system is built upon this.

The flipside of this is that running clinical trials for any kind of modification to existing devices wouldn't be ethical as well. You need to find a compromise between making it hard enough for companies not to compromise safety and easy enough for innovations to occur. Medical versions of normal items such as PCs or monitors can routinely cost 10x as much, just because the paper work to show their safety is huge.

Agreed. But then if a parent patent is recalled or otherwise labeled as dangerous, all child patents should need to reapply and cite why they shouldn’t be recalled as well.

Currently there is NO process for that. (As of the last time I watched that documentary, admittedly)

This is incredibly fascinating, and some great work. Is it possible to do the same for pharmaceuticals?

Author here, thank you!

My understanding is that with pharmaceuticals, new drugs must always go through clinical trials to prove safety and effectiveness before the FDA approves them for public use, so there's not really an equivalent to the 510(k) process for drugs.

Anyone else read this title and think of Nine Inch Nails?

Reply on news.ycombinator.com