Hacker News Reader

Comment by ekianjo | Hacker News Reader

Does the FDA conduct code reviews? And how do they guarantee that the code or the training data does not change over time without them knowing?

View on HN

Generally, the FDA (or the government body responsible for certifying medical devices) does not conduct code reviews in the sense of looking at the code and trying to find bugs.

The way it works is: the manufacturer of a medical device assesses the harm that can be caused by a software malfunction, and assigns it a safety classification (class A, B or C). Class A is used when no injury is possible, and class C is used when death or serious injury is possible (e.g. a surgical robot). The manufacturer also provides a "failure modes and effect analysis" document that looks at everything that could go wrong, what is the likelihood of the failure happening and what is the effect on the patient.

Based on the safety classification, IEC 62304 requires different levels of rigour. For example, the standard only requires blackbox testing for class B software, whereas for class C software it requires whitebox tests as well.

The manufacturer also needs to come up with a software development plan that ensures that all of the requirements of the standard are met, and an "argument" (supported by test reports, process documentation, source control history, etc) that the software was developed according to the plan.

And that is what the FDA audits: they look at the development process of a given feature and they check that the plan was followed. I think they rarely delve into the details of the implementation and are generally just checking that the safety arguments are sound and supported by evidence.

> class C software it requires whitebox tests as well

What does this mean?

test the internal components, not just the externally visible performance.

Worth noting that this is a rough description of how IEC 62304 looks at the problem, but adherence to that standard is not required by many regulatory bodies (including FDA, although there is guidance). It's a good approach to this, but there are others.

More generally, the regulatory body will be looking for you to have a formal engineering process in place and be able to demonstrate its efficacy. Part of that will be looking for how you do hazard and risk analysis, how you handle CAPA (corrective and preventative actions in FDA-speak), how you do system trace, design history file generation, etc. etc. That you have a software development plan and can demonstrate how you follow it.

So they aren't really interested in code reviews per se, but they are very interested in how you view code reviews, how you perform them when you do, what gets documented, how you perform trace an V&V etc.

Seems like they are using this to get a proper referral to a specialist rather than using this as sole diagnosis. The code itself is probably 15 lines using Tensorflow or other framework I'm guessing, but could be wrong.

If it is that simple I am wasting my time in my current job.

I got a book on TF last year and saw all the examples were fairly short and was impressed with the power. However it took me a good three months before I could reason about why reach line was there. It's pretty complex stuff that takes a while to even understand, let alone actually write.

After I wrote the above comment I remembered that just because there are few lines of code doesn't mean it's trivial to compose those actual lines.

Some of the most elegant code is terse, but it takes years of education, experience, and intelligence to be able to produce that logic.

A lot of machine learning in particular is like that. The best practitioners have very strong intuitive understandings of the data and the modeling problem, which is what allows them to construct effective models (wrt capacity, training algorithm, topology, use of certain optional training procedure additions). I would liken their job to that of a doctor writing a prescription: it doesn't take much to write the prescription itself, but it takes mastery to be able to analyze the problem and write the best prescription

The algorithm could be simple. Evaluating all failure cases, analysing them, and getting the whole lot approved certainly isn't simple.

In that case, shouldn't the training data be open source? Seems like somebody got an unfair advantage here.

In that case, the training data is the product.

They intentionally built an unfair advantage so that they could sell it.

Oddly enough that's what they're aiming to claim a monopoly over.

Here's[1] a patent they have filed towards the system. Claims 1-18 and 20 are focused on the training of the neural network. Looks like Claims 1-18 are going to be granted soon largely in that form also from looking at PAIR[2].

[1] https://patents.google.com/patent/US20160292856A1/en?q=AI,ar... [2] https://portal.uspto.gov/pair/PublicPair

They generally treat compute devices as black boxes. Performance is all that matters. But once an artifact is submitted, it's locked down. That and only that will be approved. You change a resistor, an if-then, it goes back to FDA.

What about learning? Would an update with more data and updated coefficients require going back to the FDA?

Yeah that was what I was getting at.

See other comment about code reviews.

In general though approval to market for particular indications is for one fixed configuration of a product, so your model parameters won't change.

All of this is in the process of being hashed out, but I expect for a while at least if you are doing on-line learning it will be in non-clinical configurations only and you will end up releasing an update periodically. Depending on the changes this may need a new 510(k) or not, but would definitely need a formal release.

Reply on news.ycombinator.com