Apple Watch can detect arrhythmia with 97% accuracy, study says
techcrunch.com
techcrunch.com
To the founders: what do you see as being he end game here? Are you just looking for validation (of the concept in itself, not the app- I use it on my watch)? Is the US market so different that this is particularly useful and cost effective for detection? Do you see this as eventually displaying early warning in the instance of an early warning?
Thanks and I don't mean to denegrate your efforts, but I do see lots of Consumer Med tech as solving a problem that really isn't creating value (i.e. Proliferation of devices,wearables and algorithms that proclaim the ability to help with X but are really marginally helpful at best) and I'm wondering if I'm missing something about the actual medical benefit, or whether what I feel like is true- that they aren't after being a medical device at all but instead are chasing the consumer dollars by making medical claims
The main reason is that 10% of strokes are associated with undiagnosed atrial fibrillation. The patients who present to the emergency room are a pretty biased sample--for one, they're experiencing symptoms. To prevent strokes, we need to have a way to catch AF in asymptomatic people.
Part of the challenge here is that episodes of atrial fibrillation can be infrequent--in CRYSTAL-AF, for example, it took 84 days from randomization to first episode--and existing monitoring devices like Holters or Zio patches are only worn 24 hours to 2 weeks. The great thing about Apple Watch and other consumer wearables is that they're worn for months or years. That means if we can prove the algorithm is accurate, we can get higher time-coverage than a traditional medical device, catch more AF early, and prevent those 10% of strokes described in the Circulation article.
The issue is whether the af you find through this kind of screening is associated with the same risk of stroke as conventionally diagnosed af - if not, their risk reduction from anticoagulation might not justify the bleeding risk from anticoagulating them.
The linked article suggests it might be "unethical" to even do a trial of anticoagulation vs control in af detected by such screening. That seems like a dangerous position for them to take, particularly coming from a group that is largely funded by the drug companies who sell expensive anticoagulation medication.
I would encourage readers to jump to the "sources of funding" and "disclosures" section of the linked review article to see just how many of the authors receive money from the drug companies.
None of that is a criticism of the authors of the UCSF study, this kind of technology is certainly an area worth exploring. I would just be very wary of the push by drug companies and the doctors paid by them to 1) find asymptomatic conditions 2) call it 'disease' based on older studies of patients who were much sicker 3) 'treat' them with expensive medications for the rest of their lives without actually doing research to see if it would benefit or harm the patient.
On the other hand, there is no uncertainty about the degree of risk from bleeding (small, but present, and occasionally lethal). So yes, patients with paroxysmal AF based on current technology and clinical standards benefit more from reduction in stroke risk than they lose in bleeding risk. That doesn't mean that all patients in the future, with any degree of paroxysmal AF diagnosed using more sensitive technologies will benefit similarly
I think the way my stroke colleagues look at it, is sort of the other way around - more like PAF is PAF, and as far as they can tell, based on current data, there is relatively few stratification levels that the current medical data can tell us. Meanwhile, the current assumption of risk from PAF, is so high, vs. the risk of anticoagulation (adjusted for individual patient, i.e. maybe not the patients with coagulopathies or risk of falls) that they view NOT anticoagulation as a higher risk. Because if one is wrong about it, the result is paralysis, coma or worse. If there are cheaper ways of detecting AF, all the better.
But overall, anticoagulation decisions aside, think about it - this is a potentially a cheap way to passively screen for the condition in the population. Orders of magnitude so, and possibly w/ more accuracy than a typical holter monitor or even the newer implantable devices (which in all probability use simpler rhythm analysis). Even if you are worried about the potential existence of differing levels of AF, the wealth of new data should go a long way towards further understanding AF and figuring out whether differing levels of risk exist.
Short of buying my own Holter trace machine ($500, and an inconvenient mess of wires) or having an implant (possible on the NHS, and being considered, but the waiting list is long) there's no way I can be sure if PAF is enough of a problem to justify the lifestyle changes that would be caused by going on anti-coags.
Cheap continual monitoring would be a game changer.
I can't find good tables in the article or on your teams site. [edited]
I know you guys take your numbers seriously but I'd love to see anything allowed out pre-publication.
[edit]Thanks to poster below I see this is AUC. Thanks!
1) for folks who show up in the hospital, with a stroke, often the cause is not clear. So they go home on aspirin but the latest studies seem to indicate that fully 30% of these folks with "cryptogenic stroke" end up being diagnosed with PAF after 3 months on a heart monitor.
2) this heart monitor is either a cumbersome external device they have to wear for 3 months, or the Medtronic LINQ implantable loop recorder, which is nice and under the skin, but costs a lot of money for a cardiologist to implant and monitor.
3) from a public health perspective, if the Apple Watch can automatically detect paroxysmal A-fib BEFORE a stroke (i.e. permanent paralysis, inability to speak, etc)...think of the massive societal benefits this could be...
Isn't this a version of survivorship bias? Who about to those who aren't that lucky to manage to overcome the attack and become patients...
Let me know if any of you have questions on the study, app, or deep learning algorithm. My colleague Avesh wrote a post with a little more technical detail here: https://blog.cardiogr.am/applying-artificial-intelligence-in...
Have you experimented at all with using the Apple Watch to measure blood pressure?
I have done some reading that suggests the optical sensor could measure blood pressure with some accuracy, but that Apple is hesitant to release it as a feature due to regulatory and accuracy concerns. It's my #1 wished for feature.
There was a startup that worked on using this, but they failed (due to financial but also regulatory reasons), and then remotely bricked all their sold devices...
I really wish someone would develop this further. Even if it is not as accurate as a normal measurement, the implications would be huge. There are so many people running around with hypertension who have no idea. I also don't see a risk in false positives in this case, since in principle everybody is recommended to have their blood pressure checked - false positives who go to the doctor are then just like people who read an article and go to the doctor, and are weeded out there. False negatives might be a problem - but if you don't advertise it as a blood pressure measurement tool, but just implement it as an additional warning in a smart watch, you'd reduce the false sense of security people would get if it didn't work properly.
https://www.medgadget.com/2012/01/company-claims-optical-blo...
if(TRUE) return(0)
EDIT:
I just read the headline earlier. Now after checking:
>"The study involved 6,158 participants recruited through the Cardiogram app on Apple Watch. Most of the participants in the UCSF Health eHeart study had normal EKG readings. However, 200 of them had been diagnosed with paroxysmal atrial fibrillation (an abnormal heartbeat). Engineers then trained a deep neural network to identify these abnormal heart rhythms from Apple Watch heart rate data."
So 1 - 200/6158 = 0.9675219. My method performs just as well as theirs if we round to the nearest percent. This is ridiculous.
Cardiogram engineer here. 97% accuracy refers to a c-statistic (area under the ROC curve) of 0.9740. An example operating point would be 98% sensitivity with 90% specificity.
These important details are often lost in the news. You can some more details on our findings in our blog post:
https://blog.cardiogr.am/applying-artificial-intelligence-in...
1 - 200/6158 = 97% is indeed a pretty suspicious coincidence though. I would assume/hope that they've shuffled a big dataset of recorded heart events (like the image in the TC article) and that the 200 people diagnosed with paroxysmal atrial fibrillation only rarely experience AF, so the number of true positives is probably far smaller than 1% of the dataset.
1. Even at 98% sensitivity and 90% specificity [0], which I don't think would hold up with real world usage in casual, healthy users, if AFib has a prevalence of roughly 2-3% [1] then by a quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive. With those odds, I don't think many cardiologists are going to answer the phone. You'd still need an EKG to diagnose AFib.
2. There is huge variance among people's real world use of wearable sensors, and also among the quality of the sensors. (Imagine people that wear the watch looser, sweat more, have different skin, move it around a lot, etc.) You'd likely need to do an open, third-party validation study of the accuracy of the sensors in the Apple Watch before you can expect doctors to use the data. My understanding is that the Apple Watch sensors are actually pretty good compared to other wearable sensors, but I don't know of any rigorous study of that compares them to an EKG.
3. Obviously, this is only for AFib. AFib is a sweet corner case in terms of extrapolating from heart rate to arrhythmia, because it's a rapid & irregular rhythm that probably contains some subpatterns in beats that are hard for humans to appreciate. As others—including Cardiogram themselves [2]—have pointed out previously, many serious arrhythmias are not possible to detect with only an optical heart rate sensor.
[0]: https://blog.cardiogr.am/applying-artificial-intelligence-in...
[1]: https://www.ncbi.nlm.nih.gov/pubmed/24966695
[2]: https://blog.cardiogr.am/what-do-normal-and-abnormal-heart-r...
> quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive.
For what it's worth, about 10% of people who come in to the cardiology clinic experiencing symptoms are diagnosed with an abnormal heart rhythm. So even a 20% positive predictive value would be an improvement over the status quo.
As mentioned below, you can use other risk factors (like CHA2DS2-Vasc, or even simply age) to raise the pre-test probability, and thereby control the false positive rate.
As a meta-point, I do think we let the perfect be the enemy of the good in medicine, and that potentially scares people away who could otherwise make positive contributions. For example, many of the most common screening methods in use today are simple, linear models with c-statistics below 0.8. You can build a far-from-perfect system, and still improve dramatically over how people receive healthcare today.
My overall message to machine learning practitioners sitting on the sidelines would be: please join our field. The status quo in medicine is much more primitive than we have been led to believe, and your skills can very literally save lives.
>about 10% of people who come in to the cardiology clinic experiencing symptoms are diagnosed with an abnormal heart rhythm
OK, but I'd be more careful about staying apples to apples in your comparisons; your app is about asymptomatic AFib. So how many of those people going to the cardiology clinic had undiagnosed AFib; for how many of those would a new diagnosis of AFib have changed the plan of care; etc. Kind of like robbiep was saying, I would be interested in actual added value from the larger perspective.
Totally appreciate your point about perfect being the enemy of the good. The danger is that these semi-medical wearables currently straddle a strange zone between medical and consumer use. The inevitable marketing strategy is to co-opt the positive reputation of medical products while acknowledging none of the pitfalls of consumer products. Most of the screening methods you bring up are used by a doctor on symptomatic patients with a suggestive history, and only as a partial component of clinical judgement. The way Cardiogram seems to make the most money, on the other hand, is to sell the product to asymptomatic, casual users. (Furthermore, CHA2DS2-Vasc costs 30 seconds of talking or reading a medical record, not $700 in Apple products.) So you're inevitably running up against some doubts among physicians [0].
And finally, I agree that more machine learning practitioners should join medical research. I hope the field works to set more reasonable expectations, however, as in: ML will solve very specific subtasks in clinical reasoning (as in the diabetic retinopathy study [1]). Instead, the headlines usually ratchet that up to "AI will replace radiology/cardiology/$specialty in X years." That tends to hurt the people currently in the trenches, since their contribution in bringing about practical, incremental change is diminished. The top answer of this Quora thread [2] has a good discussion of the many dimensions of the problem.
[0]: https://twitter.com/Abraham_Jacob/status/860119573915287552
[1]: http://jamanetwork.com/journals/jama/fullarticle/2588763
[2]: https://www.quora.com/Why-is-machine-learning-not-more-widel...
However, like you said, the medical field is very slow, and has quite a lot of inertia to maintain the status quo. Unless insurance companies refuse to compensate practitioners that don't use these tools, I fear that few, if any, in the healthcare field will opt to use such techniques.
And finally: How should someone with both a medical and computer science background get into ML?
This post on ML in medicine is a pretty good overview of everything that has been going on recently and the nuances often lost in the current hype: https://lukeoakdenrayner.wordpress.com/2016/11/27/do-compute...
This is a good point, and certainly nobody should go directly to a cardiologist based on these results. It seems that this would be a good system to recommend that people get an EKG done, though.
Not really, no... As you said, AFib is one of a very small number of causes of irregularly irregular heart rates (and is by far the most common). AFib is pretty easy to spot, even just by feeling someone's pulse with your fingers.
~4 false positives for each true positive.
That isn't bad, all things considered, but still a long way to go.
For reducing false positives, rather than starting with the general population, it'd be natural to start with a higher risk sub-group, e.g., people with a high CHA2DS2-Vasc score.
Earlier this week, Circulation published a review screening for atrial fibrillation: circ.ahajournals.org/content/135/19/1851.full?ijkey=StzSPk8eljGaP2G&keytype=ref
It is something like chest pain: most of the time chest pain is not a symptom of a heart attack, but it is best indicator we have so you go to the emergency room when you have chest pain. Doctors there can evaluate your situation.
On the other hand, stroke prevalence is rare, 1.3% for men, and generally don't look like mine, and my kind is 2% of all strokes, it would have been completely weird to suspect it with such a low probability and such a non-specific evidence.
Long story short, I almost got sent home with an aspirin while I had an extensive cerebral veinous thrombosis.
I suspect a lot of people don't understand how serious and crippling - mentally and physically - a bad stroke can be.
If you're lucky a bad stroke kills you. If you're not lucky you lose a good part of your brain and motor function.
In practice this means you can be left unable to move some or all of your limbs, unable to talk, unable to hear, unable to understand what's happening around you, and perhaps unable to see.
It's no exaggeration to say that it can turn life into a nightmare.
Anything that makes this less common and less likely is a good thing.
It doesn't help much though, because I don't know if its good or bad (well, actually I know but not because of the watch data). Doctors are still needed for this, and generally that includes a bunch of controlled tests and people listening to your heart while also gathering data (similarly to the watch albeit with a more precise apparatus)
I guess it can help to tell people they might wanna see a doctor if they haven't though.
I saw one story that talked about a guy whose car flipped and he was unable to reach his phone but thanks to his watch he was able to call for help.
At least I'd expect something like 99.9% accuracy (1/1000 chance of an error) when someone gives me my own heart diagnosis.
For example, the algorithm in implantable cardioverter defibrillators generates unnecessary shocks in 1 in 6 patients. Its accuracy is getting worse over time: http://www.reuters.com/article/us-untimely-jolts-idUSTRE70O7...
It captures all of these health metrics but then does absolutely nothing with it. It really is desperate for some actual killer health use cases.
We'll be incorporating these results into the app itself over time.
But as with anything in medicine... it's ready, aim, aim, aim, aim, aim... fire! :)