How we develop FDA-compliant machine learning algorithms
innolitics.com
innolitics.com
People have been shipping such systems since at least the mid 90s (e.g. CADe) but it's always been a bit of a difficult area from a regulatory point of view.
If you go back a couple decades, or really even only one depending on the panel, it was fair to describe the FDA as an organization that understood hardware but didn't really understand software. On top of this, a lot of their approach was focused on controlled, reproducible results. So anything probabilistic or even worse, continually learning, becomes difficult to frame.
The FDA guidance doc/ action plan (link is broken in article and on FDA site: it's here https://www.fda.gov/medical-devices/software-medical-device-...) is actually pretty encouraging, because they took community feedback from the original request for feedback (https://www.fda.gov/media/122535/download) and seem to have handle that process reasonably.
Many people I've talked to without direct experience seem to assume the 13485 process as a whole is needlessly bureaucratic, but in my experience it's mostly holding yourself to a higher engineering standard that some other industries, and works out ok. This willingness to engage with technologies can be frustratingly slow, but it's mostly reasonable.
I don't think 62304 is too hard to adapt; i haven't done the process work in a while but unless the latest rev radically changed things it isn't very prescriptive, not nearly as much as you suggest; I've seen agile and Agile variants done. It is a sign of the slowness of the process that it's only recently become required, and reflects 15-years ago thinking, but it's fine.
You are right that a lot of modern development processes need a fair bit of tweaking, but it's mostly because they aren't nearly careful enough for device use. This may be a bit of a burden on pure software devices, but it's probably the right trade off overall.
This is why the big shift in thinking for most developers will be the 14971 (risk management) adoption, not 62304 itself (IEC 62304 punts to ISO 14971 for this part, but it's common with hardware dev)
Your statement about 14971 being the big shift in thinking is interesting. With a bit of clinical context, do you think developers can write code with risk management in mind?
Absolutely.
Some people with more of a hardware bent are probably used to FMEA - but that requires a design to evaluate. Risk management approach (should) starts at the other end, while you are still understanding what your product should do.
If you do 14971 right (and iteratively), it will help you pin down key requirements up front that will help you avoid missteps that are costly to recover later.
One of the big shifts is that it makes you think about uncommon/incorrect usages earlier, as well as thinking about how other systems could mess with you.
Do you recommend any tools that have helped you perform risk analysis?
This document has some useful tips:
AAMI TIR45:2012 (R2018) Guidance On The Use Of AGILE Practices In The Development Of Medical Device Software
Also, we have an open source offering that includes an IEC62304 compliant software plan. You can check this out here:
https://github.com/innolitics/rdm/blob/master/rdm/init_files...
It can be a bit overwhelming, especially the first time, and especially from scratch.
I think for most people entering this space from a software point of view, the thing to get right first is some form of requirement management then later doc control. The QMS docs will come later too, but those two have the biggest impact on the engineering team itself.
For example "configuration management" was meaningless the first time I read it but really just translates to "version control"
I think having an open source example will really help the concepts make sense to engineers implementing this for the first time.
VC can be a huge part of this if you are software only, but it also helps sometimes to remember that these standards were written with hardware in mind.
Another thing that can help with understanding is to start from what they are trying to achieve (talking to someone experienced here helps) and work backwards.
For example, the idea that if I pull a version of your release software off a machine at a hospital, you should be able to demonstrate to me that you know exactly how it was produced and ended up there. The source code, the build machines, the packaging, etc. You should be able to tell me what other version exist and where they are. If I need you to notify everyone with a particular configuration of an important update you can do that.
Stuff like this leads you to the configuration management language pretty naturally.
In generally, Developing with regulatory oversight and a real QA department is just very different. The agile manifesto speaks of "working software" over "documentation". It does not speak of code. It speaks about working software. In a regulated business "working software" includes a forest worth of hopefully digital paper and the code for your customers and oversight. The paperwork is part of the output here and not part of the self inflicted documentation.
Rather, in good engineering practice the design and process, etc. is part of the product. Regulated industries just put some expectations on what that means. This can be done well or poorly, but it's basically the intent.
Don't read that the wrong way; I don't think every product benefits enough from good engineering practice to make it worthwhile - it certainly adds some friction and makes even "fast iteration" slower. On the other hand, it's absolutely the right way to approach things with high stakes for failures.
I like to conceptualize it as having a "waterfall shell with an iterative core"
It is not uncommon for us to have new requirements added in the middle of a release. However, once safety testing and clinical validation are performed, changes are pushed to the next version unless a huge safety issue is detected.
This really isn't true, at least not for this context.
As far as regulatory submissions go and the design outputs that get stored in your QMS, the SDLC appears waterfall because there are clearly defined design inputs, design outputs, and verification / validation checkpoints. In practice, however, the actual process can be agile.
I would word it differently, that the set of documents in your DHF, MDR, 510(k) filing, etc. are identical regardless of the methodology you use to create them.
Some of the language around them sounds "waterfall-y" but you are perfectly welcome to iterate on these documents. The documents themselves are more about engineering practice than process, and are an output of whatever process you use, much like the software is.
These products don't, at the moment, actually diagnose a condition or recommend a treatment which is interesting.
One of the constraints for AI treatment has always been authority and responsibility. To put it another way, if the AI completely messes up and recommends something terrible, whom do you sue?
I am hopeful that we (as a society) will sort out these issues with self-driving cars and apply some of those rules to medicine.
I hear this all the time nowadays, but everything has always been probabilistic. Most of the time, people just don't realize that all problems are fundamentally the same from the perspective of how to approach risk.
Whether physical or ML or whatever ... There's always some process that you can't characterize to your liking. So you explore mitigation options until you can make a decision about whether to accept the risk. Standards and regulations enforce that you went through those steps to get a reasonably trustworthy result.
Then liability law applies in the usual way. Actuaries and accountants tell you what it's going to cost you, and market research tells you how much it might sell for.
Inability to control or understand what you built doesn't absolve you of any fault or responsibility. All it does is change your risk acceptance criteria.
Any corporation, for-profit or not, generally seeks to obfuscate their responsibilities where possible (one could argue that this is the explicit purpose of a corporation - to be a virtual entity which cannot really be held accountable). Tobacco companies are the best and ongoing examples of this. Creating legal protection for AI such that accountability is somehow removed or diminished is not desirable in my opinion. It will lead inevitably to a situation where healthcare becomes analogous to trying to get technical support for your printer. Contrast this with the what happens currently, where you walk into a clinic or a hospital with a problem and meet a bunch of real people who listen to you. Of course there are massive problems with healthcare, and many people have unpleasant or dangerous experiences, but it can definitely be made worse if we aren't careful.
We likely won't have to deal with the legal dilemmas for a while for a simple reason: doctors _hate_ when non-doctors diagnose, and you can't build a diagnostic tool without doctors. The tools simply won't get built to any real quality standard.
There's an easier path, interestingly. It's much more effective to just give docs the info they need. A system that could do the following would be the holy grail: "Patient entered ER with [chest pain, shortness of breath], and has a history of [hypertension, diabetes]. They are on [X, Y medication], but have not filled their prescription in [Z weeks]." No diagnosis, just the right info at the right time.
They also decided it was better to have a computer make a suggestion to a doctor than to give it unvarnished to a patience, sort of assisting the doctor.
I won't disagree with experts (yourself, or them), but there is the issue of scaling it. Consider a poor person in a poor country for whom the alternative is not a delayed appointment with a real person but no advice at all.
In particular, I can relate to the author's suggestion to "visualize everything."
1. I wonder how many times the test set can be used on "incremental changes in future versions of the model" before losing statistical validity.
2. This article describes their process, but not the FDA's process. Are there specific regulatory requirements for ML models beyond their four types of reports?
2) The FDA's guidance on ML models is still in flux. Please see https://www.fda.gov/medical-devices/software-medical-device-...