Hacker News Reader

Comment by dekhn | Hacker News Reader

Hi- I used to work in regulated pharma and I want to be clear: " The core problem is that the FDA-approved devices are optimized not for getting the best result, but for ensuring the manufacturer can't be blamed if something goes wrong" is a misleading statement. The reason everything is regulated is that sometimes things do go wrong, and it's necessary to have root cause attribution so the problem can be remedied and a fix deployed, and, if somebody did something illegal or dumb, they can be sanctioned.

I wish your project success, but I think (and I've got a bit of experience) if you can't accept that regulation of health products is almost certainly necessary, and that not producing the "best" results is a reasonable if conservative approach, I will never trust your product.

View on HN

The problem is that although regulation may be necessary, the regulators incentives are different than the users. Regulators are rewarded for being overly conservative. If they are making an assessment best on the riskiness of a device versus the benefits to its users, they should say what their criteria are.

Different users are willing to accept different amounts of risk for a given quality of life improvement. Besides differing between users, the tradeoff is almost certainly in favor of accepting more risk than a regulator would approve.

I think it may help to have some specifics about what, exactly, the regulations are screwing up here.

Every type 1 diabetic has some parameters that they use to determine how much insulin they need: an insulin to carb ratio, an insulin sensitivity factor, and a basal rate (or several basal rates for different times of day). The way these parameters are traditionally managed is that your devices collect data about your carbohydrate intake and glucose at various times of day, and every 3 or 6 months you bring those devices to an endocrinologist's office where they print them out, a doctor looks at the printouts, and the doctor suggests tweaks to the parameters. This process works very poorly, first of all because 3-6 months is too infrequent, and second because calculating the correct parameter tweak requires a whole lot of math, not just looking at a printout.

The way it works in OpenAPS is that a program runs nightly, looks at historical data, figures out the correct parameter tweak, and applies it for the next day. This works much better. Much better long term health, much better quality of life, much lower short-term risk of death. But if someone dies, there's no doctor to pin it on. So device makers don't include that feature.

(This is the clearest example, but there are others as well. There are also things like where alert thresholds are set; too few alerts and you're at risk of failing to alert the user to something important, too many alerts and you cause alert fatigue and impact quality of life.)

When you say OpenAPS "works much better", while I believe that in general is probably true, how exactly do you know? Is somebody monitoring the OpenAPS fleet and its users? If something did go wrong, would you be able to recover the decision process that led to the incorrect dosing?

Deep understanding of the system, plus personal experience. There is a repository of OpenAPS users' logs, accessible to a small pool of researchers, but (for medical privacy reasons) there isn't any central fleet monitoring. Each user has detailed logs of what their own install saw and did and why, and is expected to monitor it closely. (And we do in fact monitor them--not just as a precaution, but as a necessary part of the remaining decision-making we still do ourselves. The monitoring requirement is much more manageable and less time-sensitive than it would be in the alternative, without a closed-loop system.)

I have, in fact, had incorrect dosing happen, including the worst-case scenario of an incorrect large bolus at night. (That happened because of a bug on the git head branch, not a release version, which is much riskier. As a developer of the system that's a risk I knowingly accept.)

And that's not actually that bad. The important thing to realize is that the point of comparison isn't a zero error rate, it's the error rate a human would make. The worst-case scenario is quite well understood: excess insulin, up to a configured dosage limit. And that's an error which humans definitely make--it's not even a rare mistake, it's a mistake that a human doing everything manually will make multiple times per year, every year.

When you say The worst-case scenario is quite well understood: excess insulin, up to a configured dosage limit.

Is the configuration software controlled? IE, if you had a bug in your configuration or in your microcontroller, could that bug lead to exceeding the configured dosage limit? Or is there a hard limiter (fixed amount of insulin, a physical limit switch, etc) that prevents that.

There's a software limit, selected by the user, in the microcontroller. There's a per-bolus limit, selected by the user, in the pump. There's a physical limit, in the amount of insulin that is loaded into the pump at a time (typically 2-5 days worth).

OK. Based on what you said, I see numerous situations in which a user could receive very large doses simply due to software errors. Have you had external auditors test your design from an adversarial point of view?

Well, the bloomberg article mentions that Medtronic has accepted this new uprising (which is great), but there's nothing mentioned about if the new relationship has meant the DIYers have a clear path to controlling pumps that doesn't rely on hacks.

The chances are this is unlikely, for legal reasons: Medtronic creates regulated medical devices. I wonder whether it would be possible to add something along the lines of "push this (software/hardware) button to allow non-regulated 3rd-party hardware and software to begin controlling your medical device, but doing so will bind you to <insert waiver here>" and get such functionality to pass FDA approval. I somehow suspect not, although I would be very happy to hear otherwise.

As things stand now, though, it sounds like (as the article suggests, with the picture of a "hacked pump") people are still relying on methods and techniques that it sounds like anyone can use. So yes, unfortunately everyone's in a situation where any adversary within Bluetooth range could probably take the device over, and (depending on the severity of the vulnerability being used) perhaps modify the pump itself to pump too much or too little regardless of inputs, or similar.

Quite an uncomfortable situation from a security perspective. :/

Chiming in to say that some data is available to the public as well, as these Open Humans users have shared their Nightscout data publicly: https://www.openhumans.org/api/public-data/?source=direct-sh...

These two explanations seem entirely compatible to me.

For better or for worse, in an environment where "root cause attribution" is such a preoccupation, it's not such a stretch to think that the agents acting in it would be strongly incentivized to minimize their opportunities to be the root cause - in other words, "be blamed if something goes wrong".

I don't think that needs to be interpreted as nefarious. It might just be that everyone looked at making them more automatic, and just decided that the costs to do all the research that would be necessary for CYA and regulatory purposes would be too great.

Yes. It is the "ensuring the manufacturer can't be blamed if something goes wrong" mindset that says, "it's necessary ... if somebody did something illegal or dumb, they can be sanctioned".

Another approach would be to focus on patients and ignore punishments. Which is what the hackers are doing: ignoring rewards and punishments that are driving professionals in the field.

Not necessarily even illegal or dumb.

One of the things about testing drugs and medical devices is, you really can't cover all your bases. It would be too expensive to construct a clinical trial with the statistical power necessary to reliably detect very rare side effects. I don't think anyone wants to delay releasing all new drugs to the market by 40 years so that we can be sure about what happens if you take it over a lifetime. Even post-release monitoring for these things is potentially tricky, due to the poor interoperability of EHR systems and also, in many jurisdictions, unintended consequences of health care privacy laws.

Meaning that there is no amount of due diligence that will guarantee you can't get sued. Which is fine, you can always go for proportionality instead. But then, the amount of due diligence that's necessary is driven by what costs the legal system is willing and able to impose, and, at least in the legal regime I live under (USA), they've got their own entirely other set of procedures and incentive structures that they're working with.

Insulin is incredibly expensive and is manufactured by some of the companies making these devices. I wonder how much less insulin people use with these devices?

There is no overlap between companies that manufacture insulin and companies that manufacture diabetes-related devices. (Not for any fundamental reason, it just happened to work out that way.)

Also, the amount of insulin used is not significantly different with a closed-loop system vs. not. Although closed-loop systems do increase the benefit of faster-acting insulin relative to slower insulin, and the fastest-acting insulins are more expensive (especially inside the US).

Somehow I can't help but read you've smuggled endorsement for: It's ok if the product is more dangerous when bundled with the benefit that we can blame it on some organization.

The more generous interpretation is: We value life and try to reduce the risk of severe damage when making drugs. And regulatory bodies exist to combat the incentive to just ship whatever works without adequate regard for what could go wrong

Yes, this is what I meant.

Thanks for being generous.

And what do you suggest we do when an open-source "biohack" results in something like this ?

https://en.wikipedia.org/wiki/Thalidomide

Note the mistake made, and just how subtle it was. During testing a somewhat more ad-hoc process was used to produce the drug that did not result in the bad isomere, and then it was sold with multiple isomeres in the actual drug. Regulators at the time thought that isomeres did not matter and it took years for the effects to become clear. Of course regulators did not test for things they didn't know about. Neither did the companies (not just one). Crucially the drug was NOT tested for the effect on pregnancies (and was therefore not to be taken by pregnant women, like many drugs and even some non-drug products like famously alcohol).

Furthermore: there is at least some amount of "guilt" on the part of some of the mothers: the medicine was NOT approved for use during pregnancy, and stated this.

Result: >10.000 crippled children.

Where is the guilt to be placed for this incident ? I would argue that the primary guilt is with the regulator (government) and a small amount with (most of) the mothers for taking unapproved medicine, with the rest of the mothers following doctors' prescriptions (which of course means those doctors prescribed unapproved medicine). Needless to say, the company was blamed, and forced to pay (essentially in trade for not going to prison). The German government denied any responsibility, and several other governments (famously Spain) went further: they denied anything had happened at all.

So let's say there is a timing bug in this "open source pancreas", and it starts pumping insulin into the patients bodies without cause, say at the unix epoch transition. I would guess that at least half of them won't survive.

What happens next ? Given the Thalomide incident we can make an educated guess: whatever the law says, and despite patients "accepting the risk", even when the patients explicitly make mistakes in the treatment, you can bet the government will go after whoever they can arrest, and force them to pay millions of euros, or throw them in jail for the rest of their life.

People cannot be trusted to accept responsibility for the result of picking their own treatments, even when fully informed about the risks.

I have to add: Thalidomide is still prescribed today as it is a very effective drug. Also, as a response, the US has a much stronger system for evaluating drug safety.

US regulators did not approve thalomide until after the risks were known. Their system for evaluating drug safety is the strongest in the world and that is why there were no thalomide induced birth defects in the US. Those problems, subtleties and failures of regulation were in European markets where thalomide was approved more quickly (and new manufacturing processes were less closely regulated than the drug itself). EU has stronger regulatory practices in place as a result.

The narrative is more complicated than that, see https://en.wikipedia.org/wiki/Frances_Oldham_Kelsey and https://blogs.fda.gov/fdavoice/index.php/2012/02/50-years-af.... In particular: athough Kelsey did correctly identify issues, there wasn't legal support for that kind of regulation until her work led to https://en.wikipedia.org/wiki/Kefauver_Harris_Amendment which was necessary legal support for the FDA regulation process.

True, but the point is more about that the government (and the people) went after the producer of the drug, despite that producer not really having screwed up, and left the ones that had screwed up (state, mothers and doctors) alone.

Furthermore, the law did not really support these cases. But when you cripple 10.000 babies, law does not matter. Plus we all know they went after the company, not because it was at fault, but because they could band together against it.

If that were to happen to an open source project (and that seems a reasonable guess for open source "biohacks"/medicine when they screw up) ... the authors will go to jail for life and die bankrupt. It doesn't matter what license, it doesn't matter what contracts say, it wouldn't even matter if you have personal videos of every user saying they will never hold you responsible with a paper around it signed in blood by both the user and a public notary.

THAT's why we have the pharmaceutical industry we have.

I actually don't care about anyone else's opinion of what I may do to manage my chronic, lifelong, slog of a disease. This is merely my chosen alternative to doing math and jabbing myself with a needle manually, which any reasonable person can conclude is perfectly capable of killing me within minutes. This is already delegated to all T1Ds in the normal state.

Your scary hypothetical is not possible. As if to agree, my pump just beeped at 10PM, announcing that the temporary basal (which can't kill me at maximum setting) expired and a new one was chosen. It's best choice is actually to suspend basal entirely when it knows I've dosed too much, which makes it a very legit safety improvement.

Oh, and a company created the example, which waters down your point - what convinced you that the very capable hands of people affected by the disease directly with a strong desire for data to back every thing up is anything like an accident waiting to happen? There's no profit motive here! (PS: it runs a direct test suite on install, and has cloud CI)

That's not the endorsement. The endorsement is that it's okay for a single product to be suboptimal as long as the system is optimal. It's more optimal to prevent or slow down a number of innovations in order to protect the vast majority of people from the sort of mistakes and scams that other industries accept as standard.

Reply on news.ycombinator.com