Heart rate variability differs between smartwatch and clinical testing | Hacker News Reader

Heart rate variability differs between smartwatch and clinical testing

wvutoday.wvu.edu

112 pointszdw74 comments

For what it's worth, I loathe this sort of thing (damning by vague questioning): "But, if a composite is full of a measure that’s biased, how accurate is it going to be?"

Well, I don't know, and you haven't told me. Is it a consistent high bias? Low? Variable? How biased? How accurate?

What you're trying to say, without accepting responsibility for actually saying it, is "It's an inaccurate composite because it's based on biased measurements." But you don't know that, or you'd say it without weasel words.

From the study linked [https://link.springer.com/article/10.1007/s40279-024-02066-5] (with acronym meanings added inline for readability): "The lowest pulse arrival time standard deviation (PAT σ) at 2.0 times the standard deviation corresponded to 88.4% of the Region of Practical Equivalence (ROPE) for SDNN (Standard Deviation of Normal-to-Normal intervals), and 21.4% for RMSSD (Root Mean Square of Successive Differences). As the standard deviation of PAT increases, the equivalence between photoplethysmography-derived "heart rate variability" (PRV) and electrocardiography-derived heart rate variability (HRV) decreases for both SDNN and RMSSD. The width of the highest density interval (HDI), which encompasses 95% of the posterior distribution, increases with increasing PAT σ. This increase occurs at a higher rate for RMSSD than for SDNN."

So for "how accurate?", grossly and irresponsibly oversimplifying, the Apple method is roughly 90% accurate, and devices using the inferior method are roughly 20% accurate.

I would be really interested to know of any devices not locked down to Apple ecosystem that also use this approach, if anyone has any insight.

And your comment was much more informative than the article.

That's obviously untrue

This is such a ridiculously wide divide that I'm surprised Apple hasn't used this, or something similar to this, as a marketing tactic. Imagine being able to say your medical insights are 70 percentage points more accurate than your competitors. I do understand this is apples to oranges, since these are often devices in a completely different price range, but still. I was surprised to hear this and I have been loosely following the medical wearables space for awhile.

I would imagine would be quite happy saying as little as possible about their sensors in their watches. It seems like their largest surface area for licensing litigation. Touting performance today is fuel in court tomorrow.

https://time.com/6692718/apple-watch-masimo-alivecor-patent-...

I believe there are non-Apple devices that use this approach as well, but they're certainly failing in making that information easy to find.

That being said, being the only watch with FDA blessing is a pretty effective point to market

https://www.apple.com/healthcare/apple-watch/

There are a number of watches that were given a blessing, as you put it, by the FDA. Pixel, Samsung, Garmin, Withings...

Where can one find a list of those watches/manufacturers?

Apologizes, you're correct. I vaguely remember reading they were "first" to get FDA approval for some feature, but if that were ever true, it is sorely outdated information. I fell for the hype, it seems.

Unfortunate that I can't delete or edit the post at this point.

I mean, they'll definitely market it as 4.5x more accurate. Or, for nice round numbers, 5x -- and while we're within orders of magnitude it's not unreasonable to say "up to 10x more accurate."

The Quantified Scientist has tested a while buttload of smart watches against a chest strap heart rate monitor:

https://youtube.com/@thequantifiedscientist?si=4u-u1VI7eMrXD...

Apple Watches are at the top, but there are many other watches that are almost as good.

Is there a searchable text version of this resource?

In the video description click on "Show transcript".

But it seems each video is about a different model; having to search N transcripts still isn't optimal.

He's not written books on the subject. You can setup a search yourself. There's also a blog with easier access than transcripts. https://www.robterhorst.com/post/apple-watch-ultra-2-vs-seri...

Not sure, but every video shows a huge graph with all of the watches on it, ranked by accuracy. If you just view the latest video you can get an idea of the best watches, or where a watch you might be looking at falls, then you can search for the specific video where he tests it to see specifics.

> Researchers found the method Apple uses, standard deviation of normal-to-normal, or SDNN, is the most accurate. Other brands implement a system called root mean square standard deviation, RMSSD, which Tenan said produces a wider range of error in the measurement

This is interesting for a couple of reasons. Firstly, I have one so it’s cool to know Apple designed it well in that respect, and that I shouldn’t lose confidence in the readings I get.

The second reason it’s interesting is that this kind of decision, which time-domain measure of heart rate variability metric to use, is the kind of super small design decision that really makes a system special.

Having really good developers and designers really shines through for moments like these.

That's an interesting assessment

I was curious about this and curious if the measurements made by my smart watch were "good" or "bad", and it does calculate RMSSD (allegedly)

Though it seems while they have some correlation they're all over the place https://www.researchgate.net/figure/Scatterplot-for-SDNN-A-B...

SDNN also might apply to a different time range than RMSSD

As pointed out by someone further down the thread, the value of HRV really only seems to appear in comparing to your own baseline.

Just got the latest Apple Watch, so also glad to hear they perform better on HRV. It seems they get a lot of the little details better than competitors in the wearables space. I read previously about how they likely had the most accurate sleep tracking and also had some special sauce in their GPS handling code that made it outperform Garmin.

I would love to know from somebody with an Apple watch how you use HRV.

I wear a Garmin, which I mostly like. The HRV measurement is predictive enough to be useful. Two occasions from the past 18 months are worth noting:

1) The measure dropped from green through yellow and into red. It was in yellow 7 days. On the first day it was in red, I was hit with covid symptoms. It stayed in red 2 1/2 weeks before starting to climb again; my symptoms had mostly cleared by the end of the first week, but I still felt sluggish for some time after. The kicker is that I hadn't looked at the HRV at all until sometime during the week of symptoms.

2) I went on a trip to a location +8 hours off my usual timezone, and jet lag symptoms were brutal. HRV dropped into yellow on day one of the trip, and stayed there during the 2 weeks of travel. A couple days after I returned it dropped into red, exactly coinciding with the onset of flu/covid symptoms. Again, it stayed in red throughout the illness, and then as I started to get fit again on the other side it ramped up through yellow and into green.

So even with 20% accuracy (if that is the case), it's a useful data point! What does 90% accuracy give you?

That sounds useful ! Apple have the best sensors and algorithms, but they are terrible at presenting your data.

Is this true? Do you have a source? I thought that the pros tended to make use of polar’s technology and that their heart rate monitoring (via strap) was considered the most accurate / baseline. (My sources are from what I remember dcrainmaker saying who I consider a well researched reviewer)

They are talking about wrist wearables with optical sensors, not EKG chest straps.

Though Polar’s apps are another good example of good data in a horrible app.

Pros use whomever is currently sponsoring or donating

Yeah, for heart rate, polar chest strap is most accurate. But for heart rate, running watch is almost exactly the same for me.

Others report huge differences. It could be size of wrist, or who knows what throwing off measurements in some?

Usually people will use it to see if they recover well. If your recovery is bad HRV tends to dip.

btw. in health snapshot Garmin will report both sdnn and rmssd.

Exactly my experience. After my last couple races or long workouts, my HRV dipped a few ms for a few days before coming back to normal. In this case, it’s not telling me anything I don’t already know. But in other instances, a lower HRV would make me consider slowing down a bit.

About two weeks after I started wearing my wife’s old Apple Watch, my average heart rate jumped up by ~10bpm about a week before I became sick enough to test myself for Covid, which I did indeed have!

Subsequent to getting through the symptoms my heart rate went back down to normal.

I have an Apple Watch. I don’t pay attention to my HRV on a daily basis, but if I look at it I can often see in retrospect—for example I had a cold at the beginning of the month which clearly shows up as a ~30ms drop in HRV.

Maybe I should pay more attention to it.

Adding in my anecdata:

I've been tracking HRV since 2018 (I wear an Apple Watch 24/7, aside from charging).

Over 6 years between 2018-2024, I went from being 70 lbs overweight / very out of shape, I then lost all of the weight in 12 months, started going to the gym, became physically fit, and I continue to exercise by cycling 3-4 days a week (sustained heart rate 165 for at least an hour).

Despite going from basically an overweight couch potato to someone very physically fit and strong cardio system from cycling, there has been virtually zero noticeable trend with HRV.

On the flip side, I've noticed resting heart rate correlates extremely closely with cardio fitness and stress. Resting heart rate tends to go up in stressful months or months I don't workout as much. HRV seems to just always be a flat line.

I'm also very skeptical of Vo2 max. For some reason, the watch only records maybe 2 or 3 data points per month at most. Sometimes it doesn't record anything for an entire month even though I'm doing high intensity cardio weekly. Apple rates my Vo2 max as "below average" and similar to HRV, it's a flat line. Seems to not be correlated with my perceived fitness level.

I haven't noticed any correlations with being sick. My impression of all of these metrics (including resting heart rate) is they're maybe useful, but only in the long-term to see trends over time. They aren't very useful for predicting short-term health events, in my experience.

VO2 max is only measured when you do a “brisk walk, hike or run outdoors” according to the Health app. So if you’re doing any other workout type it won’t measure this metric.

Just because it's the most "accurate" doesn'mean it's the most "useful", "correct", or "reliable" in this context. It just means reproducible, if I read it correctly.

In the same way that if they mistakenly chose to represent HRV by the heart rate, this measure would probably be very "accurate" (but ultimately false).

The issue of which HRV measure is the most useful, and what they show exactly, is still an open issue in HRV research afaiu

Apple had goals to make their these health monitoring features in their watch medically approved, to open up additional markets. For other watch makers, it was a checkbox on a feature list.

> root mean square standard deviation, RMSSD

This is incorrect. RMSSD means root mean square successive differences. Interestingly, if you actually compute what you wrote ie root mean square standard deviation, you will get the same result as sdnn.

This is why I use a chest strap heart rate monitor as a cyclist. I see the difference between when my hands are down on the handlebar. It varies around 10-15 bpm (the watch is lower than the chest strap in this scenario). Also, while hiking I see the difference after 2-3 hours. Hikers know the hands swell after 2-3 hours of hiking (you will see people raising both hands while running, that's normal) and this affects my watch heart rate calculations. There is always a discrepancy between the strap and wristwatch.

Watch tracks if my heart rate is high or not but for intense sports like mountain biking I don't expect it to be accurate at all.

I am using a Garmin Instinct 2X Solar Tactical (watch) and a Garmin heart rate chest strap.

How much is the discrepancy?

Easily 10 bpm and if it gets more intense around 150 bpm levels it is over 25 bpm. The watch is lower most of the time or delayed by 10 seconds at least. This is consistent with the 3 different brands I used. I just use the watch as a "your heart is getting faster" notification.

"But, if a composite is full of a measure that’s biased, how accurate is it going to be?"

Does it matter if the measure is biased, if you're measuring over time and looking for comparative differences?

When I read the article, this exact question came out as borne out of genuine curiosity justifying the need for measuring and comparing stuff, not a rhetorical question aiming to dismiss the thing.

And indeed, down the road, the article says:

> “There’s a lot going on between when the heart beats and when the blood gets all the way to the arm or wrist, what’s called pulse arrival time,” Tenan explained. “By simulating that and knowing what’s happening at the heart, we determined what’s being measured with the consumer wearable is different. It’s not to say the consumer wearable is bad or not useful, it’s just not the same.”

As in, electric signal ECG/EKG at chest level and blood flow PPG at finger/wrist level factually are not the same measurement, and asking "do they somehow relate in any way?"

Then the question is about the difference between RMSSD and SDNN/SDRR, and here this article (2022) explains it well:

https://tryterra.co/blog/measuring-hrv-sdnn-and-rmssd-3a9b96...

And answers TFA's question:

> “I don’t fundamentally see any reason why any of the wearable companies should be using the RMSSD measure

with:

> RMSSD is considered the best measure for short-term variations of HRV, but still a robust measure for longer-term analyses, with typical use cases for tracking stress, sickness, training, and recovery.

> SDNN is more useful for longer term cardiac health trends and analysis.

I've been tracking HRV for about 8 years and can conclude that (for me) any HRV measurement overnight using a wearable or related wearable readiness metrics are largely useless. Now, what I do - as recommended by Marco Altini, an HRV researcher - is measure in the morning, steated with a chest strap. I use his app, HRV4Training which I would recommend [1]. It's the only way to get decent data in my experience. HRV is so senstive to various stimulii you need to measure it under exacting conditions, which you don't really get when you are sleeping. Sleep stages matter, how you are breathing, body temperature, etc.

I think my Oura ring is great but... it consistently records average nighttime HRV of around 15ms, which is implausable. However, when I wake up and measure with my Polar chest strap I'll get a more sensible reading. Fully recovered it will by about 150ms. Hurting after a hard workout it will be around 40ms. Some other interesting experiences with oura/garmin/apple are:

1. Higher nightime HRV after drinking alcohol than abstaining for weeks but then I measure with a strap and it will be low as expected

2. HRV spikes when I get out of bed in the middle to the night to do childcare related things, when it should be lower when walking around

Another thing to note is that I think I have parasympathetic saturation which means due to physiology and environmental factors you actually end up with a super low HRV when you are lying down/resting. Endurance athletes often get it - I run about 100km a week and my resting HR when fully recovered is about 40 bpm, so I guess that's the reason. When I de-train then nighttime HRV goes back to normal.

If anyone is interested in HRV I definitely recommend reading Marco's blog [1], you will learn alot.

[1] https://www.hrv4training.com

[2] https://substack.com/@marcoaltini

There’s nothing new here, most athletes are aware of these issues, and there are plenty of discussions on Reddit. For instance, you can check out the Garmin subreddit. I've even had problems with fitness bands like Polar.

My recommendation is to be cautious with extreme inaccuracies and interpretations, especially for health-related reasons. It's also helpful to develop greater awareness of your body as part of your training.

I agree, and IIRC, Garmin has a warning and disclaimer in their documentation stating the HR isn't guaranteed to be accurate, and it's easy to see in a graph that it's using a moving average.

I still use mine a lot, because I'm not too worried about the exact numbers while excercising, and I feel like it's probably okay for comparing against previous activities. That's to say if the watch shows ~180 bpm for 10 minutes on a run, it's probably similar exertion to the last time I hit ~180 bpm for 10 minutes on a run, but I don't trust that it's exactly 180 bpm, and I definitely don't trust the "ramp up" and "cool down" measurements from 180 bpm.

As you said, everybody I know who cares about accuracy already uses a chest strap.

Note that the article states that they created a simulation to test. It's unclear how they validated that simulation from the abstract.

This is a weird post. I'm by no means an expert but I've read some papers on HRV and this is extremely well known in HRV research. So much so that these facts (and others) are basically copypasta at the beginning of a lot of papers. People have definitely quantified this before, like a lot. Also, the general advice is that it isn't expected to be "accurate" and the measurements are relative anyway (both to the individual and to the standard of measurement.)

There weren't a bunch of researchers (or even biohackers) walking around who thought SDNN or RMSSD could be substituted for EKG.

Sure, the statistical method matters, but I think it's missing the forest for the tree.

Any metric will differ: bpm, sleep phases, body temp, etc.

Because those devices are not super accurate, and the context you wear them in doesn't allow them to be anyway.

If you were to do a respiratory polygraph with proper medical equipment you'll find different values from your Apple Watch or Oura ring for the times you wake up. But you'll also notice a gap between getting the values at home and in a hospital, because in the hospital the nurses will check your equipment setup.

Those devices are great to give you a big picture, follow general trends and understand your own body. That's it.

I find it incredibly puzzling why Polar is left out of these conversations when it comes to wearable health tech when Polar pioneered a lot of the technology and releases white papers for all their functions.

For example, Garmin has "body battery" which is a pseudo science measure of who knows what and Polar does orthostatic tests (based on HRV). Polar won't even let you do that test with a watch, you need a chest strap because of a lot of similar reasons outlined in this paper

You cannot get HRV from an optical sensor accurately.

Not even at idle, but definitely useless in motion.

And definitely not at the wrist.

Optical at the wrist is all manipulated data via the watch software that is guessing what it is seeing or not.

The only real way is via a heartrate strap that is electrical.

And the next generation of sensors is going to be even more amazing with EKG

https://www.movesense.com/movesense-flash/

Seems odd to compare a machine like a 12 lead EKG to a smartwatch with a single lead EKG support.

Of course variability will differ, that’s why they invest to much into improving the technology and ml algorithms to make up for the difference to personalize the data.

The watch could encourage you to go to a doctor and the doctor can use a more accurate instrument to diagnose.

If you do Apple’s mindfulness breathing exercise it always takes a reading after that which tends to be closer to an accurate HRV.

How do you know?

Health > Heart Rate Variability > Show All Data lets you see the exact timestamp and source for any stored data point. And for HRV specifically it will give you the list of millisecond-timestamped heartbeats it considered for that measurement as well.

But how do you know that this is (more) accurate?

Because HRV is the most reliable after parasympathetic activation in a nutshell. You can look it up for a better explanation.

O’rly????

Every owner of Garmin I know complains of the absolute useless POS it is to rely on the optical HR for HRV. A HRM on a strap will get you good/very good readings for HRV. Finally there is a good podcast from Attia which he talks with an HRV expert behind an HRV dedicated product.

I eventually stopped wearing my Coros running watch because I started to not trust the heart rate thing. Whenever it got cold, the readings shot up! Now I'm just trying to train myself to remember the right tempo from touch.

we had patients come in complaining their smart watch had high readings. after testing they usually had something unusual or indicative. but they were also large... or had other signs.

running an EKG is cheap and easy (might not be cheap to pay for, but it's not an MRI!). i still like patients having more access to health data - legibly - for them. if they were doctors they'd see signs by skin health, etc. the watchs are overall a net positive imo.

.mris are cheap too but a headache to schedule cause everyone needs the machine and there might only be 1 or 2 in a hospital.

I don't like using smartwatches for HR tracking during exercise. If for any reason blood flow to your wrist decreases it becomes inaccurate

But when it's inaccurate then it's usualy blatantly obvious. For example if you are out on an easy jog and it shows 188 bpm then that's your cadence, not your heart, unless something is going down really badly and then you don't need the watch to tell you that.

It's not like it will show 147 when the actual value is 134.

you have to measure during periods of deep sleep to get accurate results

this is why you a whoop, not an apple watch

I don’t follow your reasoning here, what’s the Whoop got that the Apple Watch does not that’s relevant here?

The whoop measures and averages your HRV over your deep sleep to give you one number.

The AppleWatch, at last time I researched, just gives you your HRV, NOW. And my impression is it's not really a useful number as a snapshot, since it varies so much during the day it's not really useful to know what it is at 10:40 AM EST.

My Apple Watch appears to have taken HRV measurements at 2 hour intervals last night overnight — is it a granularity difference then?

I think the idea is it has to during your deep sleep specifically, which may or may not overlap with those 2 hour intervals.

It's hard to understand how Apple, one of the most valuable companies in the world who employs top-tier chip designers, hardware and software integrators, is unable to do something (measure HRV all day/night long) that its competitors do, in a device whose battery lasts a fraction of Whoop and Garmin.

because they are not focused on the small details. Whoop is so focused, they don't even have time to add a clock to the wearable

i have wildly high HRV and my cardiologist says it doesn't really mean anything

815 ms average HRV when normal is like 50-200 ms

Everyone has a different baseline. For athletic training and recovery they usually look at variations from the baseline rather than the absolute value.

Wouldn't this be closer to arrhythmia then?

Technically, any value > 0 is ‘closer to arrythmia’, but yeah that is a pretty high value. 60 bpm == 1 beat per second (if everything is perfect), so if the HRV is 850 ms and their heart rate is around 60 bpm that is what - 85% variation?

If that is ‘consistent’, I’m not even sure what that EKG would even look like, but ‘regular’ seems unlikely. I’m honestly not even actually sure what ‘beat’ would mean when it’s > 50% variation.

So can we just have our wearables give us both calculations? I don't mind breaking HRV into two measurements, SDNN and RMSSD.

Heart rate variability is just normal sinus arrhythmia that varies with inspiration and expiration. The relative variation is higher with a lower heart rate… I don’t think there is any independent information in it aside from lower heart rate in endurance athletes…

Reply on news.ycombinator.com