The Apple Watch heart monitor sends too many people to the doctor
theverge.com
theverge.com
I... utterly fail to see the problem here.
If it were 0.01% that ended up having a problem, then OK I can see how this is not a signal worth paying attention to. But 10%?! 10% is huge.
Plus, there's no evidence of any harmful side effects here -- it's not like people are being sent into unnecessary surgery because of what their Apple Watch said. They're simply being given normal tests.
Now if Apple can make this more accurate, then great. But if this saves even a few lives, then the fact that 9 out of 10 warnings are false positives still seems extremely worth it, no?
The article paints this as a failure, when all I'm seeing is success.
No, not by a long shot, because physicians are a pretty rare commodity, care is expensive and sending nine out of ten people to the doctor for no reason is a huge drain on the system and public healthcare resources.
if you see this as success I don't think you're familiar with the concept of opportunity costs. For people who don't have any symptoms, have no history of heart conditions or other underlying risks, running around with an EKG on your wrist is absolutely ridiculous. All it will do is inflate healthcare costs with little improvement in outcome, which is already the primary disease of the US healthcare sector. And of course apple's and the healthcare sectors primary motivation to sell these things to begin with.
Not every cardiac condition even requires immediate attention by a physician, I wager most of the anomalies that show in the 10% are benign arrhythmias that would have been picked up on a yearly routine checkup in due time.
I don’t think this is anywhere near enough of an impact in terms of “false positives” to be a net negative.
10% positive predictive value for something like this is pretty good. It’s certainly better than “hey look at this weird mole.”
That's not happening. Only a small number of people went to the doctor. Of that number, 9 out of 10 ended up not being diagnosed with a (new) problem, but it seems that even some of them had related problems that were already diagnosed.
Regardless, we don't know the false positive rate. It's not 90%. You need to know the number of negatives before you can calculate it.
> No, not by a long shot, because... [opportunity costs, overloading the healthcare system]
Totally agree with all of that.
Why don't EKG salons exist (like nail and hair salons)? New market opportunity created by the Apple Watch?
In this case, all we know is they had they had a “clinically actionable cardiovascular diagnosis”. That action might have saved lives or might have added a few healthy months to the lives of a few, at the cost of requiring 20 to undergo treatment, spend time waiting for treatment, be worried, etc.
There can be a point where the cost and risks of testing people we can’t make healthier (either because they aren’t ill or because they are, but no effective treatment exists) outweighs the gains made by treating people we can treat.
That’s what the article being discussed, IMO rightfully, tries to measure.
A new device is created that alerts 50% of the US population that they have a condition that needs to be treated. Only 10% of those cases warranted further action.
Do you imagine there would be adverse side effects of hospitals being overrun with patients that didn't need to be there?
How can you utterly fail to see a problem with an influx of false positives to an already overloaded healthcare system?
I really have distaste for those comments that attempt to mask ideology with a veneer of logic. You attempt to use a limiting case argument that actually proves the original comment's argument and then end with rhetoric about how the healthcare system is being flooded already. That may be true, but that isn't relevant to the utility of the apple watch in this scenario.
Since we're overly simplifying, there _are_ devices that can detect conditions that 5% of the population has _and_ have far fewer false positives - X-rays, CT scans, MRI. In the grand scheme amazing things, but no-one's calling them "miracle devices".
Perhaps, but you can't wear one on your wrist and you generally have to be quite sure something is wrong first.
What would be more likely to happen in that scenario, is that so many people would be seeking care that the 5% who were positive would be unlikely to even get the care they need.
Further, such an influx would have knock-on effects on unrelated patients, whose care would suffer due to the severe lack of resources.
This is all magnified if the resource is not a general practice physician, but a specialist.
If the healthcare system - paid healthcare system - can't handle a slow baseline increase in non-emergency visits, then something is seriously wrong with this picture.
Hypothetically, let's assume over a long period of time new devices are alerting users to possible medical conditions. Those appointments are paid for (I'm guessing). Will that money not fund more availability over time? The supply of doctors is not supposed to be fixed.
Out of that haystack, only 41 referred to an Apple Watch alert about abnormal pulse detection.
And of that 7 were found to have enough concern to further explore potential cardiovascular disease.
So, out of three quarters of a million visits, there were 41 with concerns from an Apple Watch, and 7 people received early treatment they might not have otherwise received (which could potentially save millions of dollars of future healthcare costs).
34 people were false positives and inconvenienced with a doctor's visit.
This seems like a net positive.
Suppose a device is released that has a 30% false positive rate: two-thirds of the people it identifies have an actual underlying condition that requires treatment.
Presumably you would not be opposed to this. (If you do, you'd seem to be opposed to any effort to catch conditions early.)
So then we're at a discussion about what the correct percentage of false positives is for a further diagnostic - in this case, likely a holter monitor, to identify if it's an actual issue, which is not that expensive.
You believe it's obvious that ten percent is an outrageous number. But if the next step is a fairly inexpensive diagnostic, and the subject matter is a potentially lethal condition (often the case with hearts), it's not obvious at all.
What's your acceptable "brightline" of maximum false positivity to warrant additional diagnostic? Fifty percent? Twenty percent?
But what you described sounds reminiscent of the PCR test we have for COVID right now. I at least hope many are aware of this here.
It’s off topic, but I recommend you do it. I’ve found HN to be a great place for those types of discussions provided you have a relatively thick skin and don’t engage in hostility.
This just improved the lives of 16 million people. That is not insignificant. And that's assuming 16 million people would have the same condition.
This also might be underestimating the number of false positives doctors already deal with and that these devices may actually lower that number.
Here's the numbers from the study:
https://academic.oup.com/view-large/figure/207880397/ocaa137...
Full study:
Doctors will rarely subject anyone to thorough evaluation for things if they aren't over the common danger age and don't report specific symptoms. Your typical annual checkup for people not 50+ is a fairly generic blood panel and very little else. After 50 they may add, what, a prostate probe for men?
I see a large part of a doctor’s job as being an expert consultant for an individual to make decisions about their own healthcare. Someone who is interested in and involved in their own risks is likely an extreme outlier to begin with.
In my life I've had one doctor who performed an EKG as part of my annual checkup, once.
The other ~10 primary care physicians I've had never did -- my annual was mostly just a few questions, some blood work, quick reflex tests and I think they looked in my ears and eyes.
If my Apple Watch detected a heart irregularity, I can say with certainty it's not something that would be caught otherwise at my next annual checkup.
I personally got a false alert once. I kept a eye on it, and it was fine - mentioned it my doctor, did a EKG just to be sure at my next checkup.
That's the way this should work.
Arguably this allows Mayo to support lower income people but medical over-utilization is a part of Mayo’s business model.
> The Apple Watch abnormal pulse detection feature was designed to address asymptomatic (so-called silent) atrial fibrillation. Atrial fibrillation is the most common cardiac arrhythmia, affecting over 30 million people worldwide[0].
So, assuming there's 8 billion people in the world, this affects .375% of the population. Assuming this was a statistically significant study, that means that if everyone in the world had an apple watch, this would only let 300 million people know they potentially had an AFIB event. Now, an excess of 270 million might seem excessive--but I would weigh excessive with the burden this event puts on the medical community. The primary means of diagnoses for an AFIB event is an ECG[1], which costs $150 on Amazon. Assuming they don't have to buy an ECG for each individual, I would assume the cost to most health care clinics for diagnosis is around $100. However, this test
1. Does not have to be administered by anyone other than a nurse. 2. Only has to be administered once in a lifetime for a diagnosis.
So now 10% doesn't seem that bad given that the burden is distributed to the global healthcare system over time. Keep in mind too, even if every single one of these people visited the ER, then this would be completely in line with the fact that only 10% of ER visits result in an admission[2] to the hospital.
[0] https://academic.oup.com/jamia/article/27/9/1359/5911974
[1] https://www.mayoclinic.org/diseases-conditions/atrial-fibril...
[2] https://www.cdc.gov/nchs/fastats/emergency-department.htm
That's really the doctor's issue to address, not the device's. Sticking your head in the sand isn't the right solution to the problems of dealing with information too aggressively.
> They've backed away from screening as aggressively for certain cancers because the risks of aggressive treatment turned out to be significant and it turned out that more people than expected had tumors which didn't develop into a problem before they died of something else.
Literally the only way to build viable metrics for which tumors will develop into problems is for _everyone_ get screened, not for no one to get screened. The problem of treating too aggressively is a treatment problem, not a measurement problem. If we avoid measuring, we have no way to learn how to treat better.
If we decide to never look because we don't yet know how to handle the information, then we will never know how to handle the information.
One of the most important questions you can ask your doctor is "what happens if I don't have this test?" and "What happens if I don't have this treatment?" But most people don't ask these questions.
And so it really is a measurement problem. Over-testing, over-diagnosis, and over-treatment are well understood mechanisms of harm.
https://ebm.bmj.com/content/23/1/1
https://www.bmj.com/content/362/bmj.k2820
The harm caused by inappropriate screening is so severe that we (UK) have a committee to weigh up the harm vs benefit of different screening programmes. https://www.gov.uk/government/groups/uk-national-screening-c...
Putting the burden of asking these questions on the patient is wrong because patients don't go through a decade of training to become patients. And expecting the doctors to always ask these questions is wrong because I guess now we're not supposed to trust our doctors. But that doesn't mean we should suddenly live in a world without patient advocacy. It means you need to put it in the right place, in the hands of neither the patient nor the doctor directly. But if you never measure, and you need to be clear that, modulo epsilon, nobody ever measures heart rhythms despite heart disease being the #1 cause of death, then you definitely have no chance for anyone to ask those questions.
> over-diagnosis and over-treatment are well understood mechanisms of harm.
So are under-diagnosis and under-treatment. Both can only ultimately be solved by more information, not less.
Again, it's not that simple. Doctors do not have perfect knowledge and have factors to consider other than the current academic consensus. Some fraction of people will get scared and demand _something_, which will for some people lead to them doing something worse than nothing at all. If you have many people seeking out treatment for a notification which has them scared, some of the patients will try something other than good medical treatments and some of them will have doctors who recommend things which used to be or are no longer best practice, make a recommendation skewed by a financial incentive or what will turn out to be a biased study funded by a company with a financial stake, etc.
Engineers building tools like this have a tricky ethical line to walk between making information available and considering how that information could be misinterpreted or faulty. One of the biggest things I'd want to see on a system like this is careful treatment of showing confidence levels and present them in the UX. “️ Heart anomaly detected!” is not the same as “Our sensor reading was unusual but the signal was low-quality. Do you feel okay?”, etc.
https://www.google.com/amp/s/www.cnbc.com/amp/2018/02/22/med...
Damned if you do, Damned if you don't.
For example, "One example of a lethal error of communication between provider and patient occurred when cardiologists failed to warn their 19-year-old patient not to run. The patient had experienced syncope while running, and 5 days of inpatient, diagnostic testing were inconclusive; however, his cardiologists knew he was not ready to return to running but failed to warn him against this risk. Having not been warned against running, he resumed running and died 3 weeks later while running." https://journals.lww.com/journalpatientsafety/Fulltext/2013/...
This categorization assigns 100% of the blame to a cardiologist who was not performing a procedure and who did not know that the patient would die. The patient chose to do something for which they'd recently experienced issues with, without waiting for complete test results or clear instructions. It's an accident, and unfortunate, but my point is it's very problematic to conclude that the cause of death here is a medical mistake, rather than a heart condition. This is the representative example that the Journal of Patient Safety uses to illustrate communication errors, so it seems reasonable to assume there are many such examples.
Furthermore, how many people would be dying if there was no care instead? Probably a lot more, and that's not accounted for. To call mistakes the 3rd leading cause of death, I think we need to account for how many people were saved by medical care too... and when you think about it that way I suspect it's complicated if not impossible to tally that number, meaning that it's also complicated if not impossible to rank medical accidents as a cause of death against objectively identifiable categories like heart attack vs car accident.
Following your logic here, if you have a toddler and a bucket at home, it would reduce your toddler’s overall risk profile if you bought a gun and shot holes in the bucket.
In other words, there’s a lot more that goes into using statistics to inform behavior than merely seeing what group you happen to be included in for a single breakdown.