Fitness Trackers Are Only 67% Accurate, New Research Finds | Hacker News Reader

Fitness Trackers Are Only 67% Accurate, New Research Finds

wellnesspulse.com

75 pointsnabla983 comments

Why are they reporting correlations as percentages? That's incredibly misleading. If one wants a more intuitive measure of concordance than the correlation, one could use the coefficient of determination, which is the correlation squared.

This number tells you the percentage of the variation that is accounted for by the tracker. For a deeper introduction to correlations, see e.g. [1].

Let's rewrite the summary with the correct figures (and reduce the ridiculous number of sigfigs):

> The analysis reveals that commonly worn fitness trackers are the most accurate for measuring heart rate, providing a strong accuracy of 58 %. However, they are only moderately accurate for tracking step count and energy expenditure, 47 % and 32 %, respectively.

The highest level of accuracy is provided by the Apple Watch, which exhibits 74 % accuracy for heart rate and 50 % for energy expenditure. Garmin is the most accurate for tracking step count, as it offers 68 % accuracy.

----

[1]: https://entropicthoughts.com/the-surprising-richness-of-corr...

> When it comes to energy expenditure, Apple is the only fitness tracker providing a strong accuracy of 71.02%. Jawbone, Fitbit, and Polar yield only moderate accuracy, ranging between 50.23% and 65.57%.

I don't buy it, no brand can be accurate on energy expenditure because it depends on a lot more things than what the watch can monitor.

Even in lab they have to put big mask on people while exercising to get something somewhat accurate.

On watch, calorie counter is a function of time and heart rate. Yet most watch brand can't even get heart rate right beside apple and Huawei.

Tried all the brand, went back to using a 10€ Casio and put a H10 monitor when doing sport.

>I don't buy it, no brand can be accurate on energy expenditure because it depends on a lot more things than what the watch can monitor

To be fair, the comment you excerpted noted that the best performer was still just 71% accurate. So they’re saying they’re far from perfect.

>On watch, calorie counter is a function of time and heart rate.

Well, we just don’t know. But let’s take Apple as an example. Apple knows your workout type. And they know your weight if you’ve entered it. And lots of other sensors as well (O2, if possible, temp, accelerometers, etc).

They’ve put a crazy amount of money into their testing lab where they do have extensive metabolic rate testers, VO2 Max, etc.

Will they get as good as that equipment for a dinky watch? No. But I also am glad I don’t have to strap a face mask on during my daily bike rides.

Can they, with insane volumes of data and hard research, make pretty good directional inferences using models that match their watch sensors to gold standard tests? It seems the answer to this (both logically and in the research data) is yes.

This video is a decade old so they’ve only become more advanced and have more data since then, but there are lots of models you can build to power these estimates: https://youtu.be/BceaTNT14Ao

I’m glad you’re happy with your Casio. Keep rocking what works for you. But I’m not sure what’s so hard to believe here when other researchers have actually produced the data on their accuracy.

At least in the case of Apple I have met a couple people who work on the fitness products and their accuracy. One of the people does real world testing so they are diving one day and coding the next. Great work if you can get it and your hobbies align.

The lab setup you describe is basically what they have to setup to correlate their proxies to energy. I’d expect the variability is going to come as a factor of body efficacy.

I used to be in much better shape (resting heart rate in the 50s) but I’m not much heavier and haven’t lost much of the muscle I gained (70ish percent of my former one rep max achievable). My day to day non-workout energy burn is measured higher now than it was when my heart rate was lower for all activity levels. I don’t think that it’s true I’m suddenly +15-20% energy consumption simply because I put on 10lbs but rather having a 10bpm increase in heart rate is probably what is being measured.

Short of that though it does a pretty good job. If I’m doing things that are strenuous but my arms are stationary it picks up the work. Very old wrist trackers needed to have an arm pumping or otherwise moving to do anything aside from a pre-calculated “you are alive” burn.

Given the level of effort for what we get, I’ll take it as a basic way to answer “did I actually get some exertion from watching my 2 year old today?”

> Given the level of effort for what we get, I’ll take it as a basic way to answer “did I actually get some exertion from watching my 2 year old today?”

That's what I initially started using it for but the best finally is just to do exercise everyday. Not everyone has time to run 2 hours or bike during 3 hours, but the bare minimum to stay healthy is 30 min of moderate to high intensity workout, which I think everyone could do.

If you stick to that, there isn't even any need to monitor yourself with a smartwatch.

> I don't buy it, no brand can be accurate on energy expenditure because it depends on a lot more things than what the watch can monitor.

You could say the same about virtually every modeling problem, yet the world runs on models that work adequately in practice. You measure what you can, infer what you can, and fill in the gaps with priors.

As the saying goes: all models are wrong, some are useful

Not adding much to the conversation, but Google Pixel 3 is also quite accurate during running and cycling in terms of HR measurements, according to the already mentioned YouTuber 'Quantified Scientist'.

> no brand can be accurate on energy expenditure because it depends on a lot more things than what the watch can monitor.

I was going to quip that I can probably look at a person and state their energy expenditure with 70 % accuracy (whatever that means -- I'm going to assume coefficient of determination.)

Just someone's weight alone ought to have nearly a 70 % r² with energy expenditure. (A quick internet search suggests 71 % though I have not bothered verifying their methodology.[1])

Of course, they could mean 70 % across days for the same person, which is significantly more impressive as it takes out the strongest signals, but it's not at all clear what they actually mean and that's sort of my point.

Edit: I read the article closer now and when they say 70 % they actually mean a Pearson r of 0.7, i.e. a coefficient of determination of 50 %. Reporting r as a percentage seems confused at best, or possibly dishonest.

[1]: https://www.researchgate.net/figure/The-relation-between-bod...

All reasonable fitness trackers use your weight as an input, and there's nothing in the article to suggest the devices weren't provided with this information even though it's not "something the watch can monitor".

That's what I'm saying. If we're discussing an r² of 70 % on a population level they don't need to track anything to achieve that, just use the weight. So before we can say whether 70 % is good or bad we need to know more about what those 70 % actually mean.

On top of this inaccuracy in actual measurements, many fitness tracker brands invent their own metrics.

"How did you sleep?" I asked a friend who stayed over, - "I don't know" they replied as they raise their smartwatch to their face. "Pretty well, apparently! I got a sleep score of 84."

I've also heard of an acquaintance that they cancelled plans because their (Garmin™) Body Battery was too low.

Instinctively I flinch whenever people place such importance in what is essentially an arbitrary techno-horoscope. But I also find it hard to make an argument against it as long as it inspires healthy behaviour.

For a long time I was sort of baffled by the (bad) level-of-fitness and restedness metrics I was getting in a couple of popular apps.

At some point I decided to figure out what the deal was with it, and I saw that their calculations were based on resting heart rate, but because I only ever wear my Garmin running watch during actual runs, it was inferring my RHR from the lowest HR it picked up before or after my run, and reporting to every connected app that mine was like 120bpm (my real RHR is more like 50bpm). Everything was seemingly happy to just accept the garbage data and plug it into their formulas to produce garbage output. Really they probably should have just told me to see a doctor if they actually believed I had a 120bpm resting heart rate!

After that I turned off all of the automated metrics—it was a big realization about how easily messed up they can get (even if you wear your watch most of the day, you might have a fairly different RHR based on whether or not you sleep with it on). I can see this exact situation causing someone a lot of undue anxiety.

>I've also heard of an acquaintance that they cancelled plans because their (Garmin™) Body Battery was too low.

I wouldn't take that at face value. Sounds far more likely that they already wanted to flake and needed an excuse.

Yep. I expect they were feeling tired and the smartwatch readings just confirmed it. If they really felt like coming, the readings wouldn't stop them.

> I've also heard of an acquaintance that they cancelled plans because their (Garmin™) Body Battery was too low.

This sounds like someone who just feels like they need to provide tangible evidence for their reason being "I feel terrible"

The garmin stuff kind of works as a realitive measure when most things are constant but completely break if you do something unusual like a long hike or a new cardio regiment

Well, sleep score still provides value to those with zero curiosity.

For everyone else, you can click into sleep score and see how much you actually slept vs think you slept which is useful data to anyone who wants to take sleep seriously.

Health wearables give you some accountability.

I wear my Garmin 24/7 and generally trust it. I wouldn't cancel plans because of what it said but I can definitely tell when it's showing me data that reinforces how I feel (good or bad).

There is a YouTube channel named "The Quantified Scientist" which offers technical and more in-depth reviews of varied fitness trackers: https://www.youtube.com/@TheQuantifiedScientist. I would very much recommend it if you intend to buy a wearable and care about accuracy

I love his channel. The TL;DR of it is that Apple Watch generally has the best overall health accuracy when compared to other wearables.

That depends on what you're looking for. Apple Watch has a pretty good optical heart rate sensor but it's not great. Everyone knows that if you want accurate heart rate data you need to use a chest strap.

Other data point: he wears the Whoop as his daily wearable.

For the top quintile of gadgets, accuracy isn't what sets them apart but rather everything else (battery life, UI, data presentation, form factor, etc).

As long as any error is consistent, I don't really mind. Mostly I use it as a way to track progress over time anyway.

Exactly. It doesn't really matter.

If your tracker shows you walked 12000 steps today as opposed to 8000 yesterday, even if you actually walked 10500 today and 8750 yesterday, that's still progress.

Yeah. I have noticed that the relative changes in numbers seem to be very correct. I can’t validate the absolute numbers, so I don’t really look at those so much. For the Apple Watch, I’ve noticed that the sleep tracker is quite accurate too, in absolute terms. That is, when I remember having long, wild dreams, I have a substantial amount of REM time. When I don’t remember my partner doing whatever at some hour, I was in deep sleep. If I toss and turn a lot, I see a lot of awake segments. So that part seems pretty accurate.

I recently started using one of these. The other day, I was working out on a rowing machine and received an alert (twice!) that my heart rate was extremely high. I was surprised because I wasn't even pushing myself that hard, and I am a fit person in general.

Upon research and asking around, I understood that it's more of a false positive. A more paranoid person may have reacted differently in such a scenario. So, consistency alone may not always suffice.

You need the band on your tracker to be very snug or this is likely to happen. You're actually seeing the tracker move, not the heart rate.

I've seen substantial mismatches between wrist and chest strap if my tracker had any ability to wiggle. With an infinitely adjustable metal band set so there's no slack but no pressure I see a good correspondence, although I haven't put it a test as hard as rowing (my activity of choice is hiking.) But my wife absolutely doesn't like such a band, wearing a soft, loose band and using her arms her "heart rate" has gone above 200.

I have also seen an environmental effect--these days I use the chest strap sensor because if my wrist is cold (conditions just below when I would add another layer) the wrist data can go totally bonkers.

That's a good point, thanks.

Wrist based heart rate accuracy during rowing and cycling is notoriously bad.

All the fitness super-nerds have been using chest straps long before wearables were a thing.

For the heart such errors are problematic

From what I gather, for fitness and on an individual level, the delta seems more interesting than the current rate.

Yeah. They may not be terribly accurate but if they are precise they can still be useful.

That's not how error percentage works and why do we all just accept that everything has become mediocre?

In what fantasy past has cheap, personal activity tracking a) existed and b) been better?

Could it be better still? Sure, but I see no reason to assume that people are somehow lowering standards and not working on making it both cheaper and better all the time.

The fact that a technology hasn't existed before is not an excuse to sell that technology with an implementation that doesn't work.

Well we're discussing whether this constitutes "doesn't work [at all]" or "doesn't work [perfectly]."

I am even OK with selling, but they should drop the usual PR lies and be honest. No business will ever do that though, consumers in general are not that smart/ have good critical thinking.

So I don't see an easy solution apart from governmental regulation. But you need a consistently stellar government for that.

There are multiple meanings of the word "error".

I just don't care how accurate that the watch is (thus far anyway) because the perception of progress—delusional or no—is why I use it.

Generally, though, my perceived performance is reflected in the relative numbers. If I felt otherwise I too would be complaining about accuracy.

I use an Apple Watch, and while I don’t necessarily care about its accuracy in terms of steps walked or other specific details, I do value the insights it gives me into my daily activity. Even if it’s not entirely accurate, I get a reasonable estimation of how much I’ve moved, stood, and worked out. My watch motives me to me more active.

True. I think most people care more about trend lines than anything else. If it isn't accurate, at least let it be consistently inaccurate.

I like to say that I don't care if it's inaccurate, as long as it's precise.

FYI, precision is an entirely unrelated to accuracy, whether consistent or not.

Precision is basically just how many decimals your measurements have.

Trend lines are way more important for pretty much everyone. Anyone who cares about absolute numbers is either focusing on the wrong thing, or is doing vo2 max tests on the regular and knows the numbers on their watch are trends anyway

Disclaimer at the bottom of the article:

"The evaluations presented are based on general observations and should not be interpreted as definitive assessments of specific brands or their features."

The most recent paper used for this meta-study is still pretty old, and this technology changes. The study tells you almost nothing that could inform a buying decision regarding today's Apple Watch vs. Garmin Epix Pro (as examples). The sensors on the market today didn't exist and were not evaluated in any of the papers.

If anyone like me was wondering how accuracy of a continuous measurement can be a percentage, apparently they're just multiplying correlation by 100.

> Correlation coefficient estimates and their 95% confidence intervals were also expressed in percentages for clarity and readability.

Can someone help me understand the percentages? Does 80% accurate imply that the measurement could be 20% inflated?

I'm genuinely surprised the accuracy is that high.

Agreed. If you want something more accurate you are going to need something more bulky and invasive. Like a heart rate monitor chest strap.

It's simply a trade off between convenience/cost and measurement quality.

67% is good enough for me. I am not using it as a medical device, and I am not an elite athlete. I just enjoy doing exercise, and every now and again I like to review the stats from my watch.

It's like the fat % on my bathroom scale. I know it's not perfect (in fact body % measuring is incredibly hard with massive error margins even in the most expensive/intrusive tests) but I'm not looking for exacts, I'm looking for trends and I've seen them where I expected to. So yes, the % may be bogus but I can use all that data to see trends which is all that matters (to me).

precision != accuracy, correlation != accuracy

let f be a continuous real valued function, let c be a fixed bias constant, let pcc be the pearson's correlation coefficient function.

pcc(f) = pcc(f + c)

f + c is precise, but not accurate

The use of correlation was bothering me too. If your watch always read 2x your actual heartrate, it would have a 100% accuracy under this metric.

Do you mean 100% precision?

>The analysis reveals that commonly worn fitness trackers are the most accurate for measuring heart rate, providing a strong accuracy of 76.35%. However, they are only moderately accurate for tracking step count and energy expenditure, 68.75% and 56.63%, respectively.

>The highest level of accuracy is provided by the Apple Watch, which exhibits 86.31% accuracy for heart rate and 71.02% for energy expenditure. Garmin is the most accurate for tracking step count, as it offers 82.58% accuracy.

Not surprised, it’s a shame the folks who obsess over the numbers from these devices will be the ones least likely to understand what this actually means though.

Seems like HR evaluated here was just from the wrist device (like watches) and not HR strap. Is that correct?

(I only track HR with strap and power on my bike power-meter).

Yeah they’re not considering straps and arm-worn optical sensors here, just the built in optical sensors.

Why didn't they include Whoop or Oura?

I've had an iPhone for many years and just recently discovered the Health app.

My phone has a lot of my "steps", an estimate of my calories, data about my walking stability, and tons of other stuff. You don't even need a watch to get +/- 65% accuracy on anything except heart rate.

I remember one time when I got my daily step counter alert when I was eating chips on a sofa. I chuckled.

For step counting they really seem to only track certain hand motions, I though they would somehow notice how feet hits the ground and vibrations travel through the body to the wrist.

Phone app tracking distance, spoke up "4 miles". While I was sitting eating lunch. (Canyon problems, the GPS fix kept bouncing around.)

AI assisted meta analysis. Just piling on more exercise science trash.

That's better than all SEO tools and probably most analytics imo (only measured SEO tools myself) ... the value in tracking is mainly in spotting trends.

I wish they included Oura in the research.

Do they list the models of the brands they are talking about anywhere? For example, they mention Garmin but there is a huge variety of Garmin models with varying accuracy. I'm surprised I can't find it anywhere. They mention gathering data ("The majority of commercially available devices measure these basic metrics, while other metrics typically vary between different brands and models.") on different models but then group them together under a single brand? Why? Frankly, this analysis seems a bit half-assed.

Are the phones more accurate?

No.

I find fitness trackers/smartwatches expensive, useless, and frankly, creepy.

Why do I need to pay $400 for an Apple Watch that sometimes doesn't have enough battery to make it through the day, and aggressively insists on tracking every movement of mine on the grounds of 'fitness'?

Seriously, that 10 hour stand goal and 'streaks' and what other achievements are just to make sure this GPS and heart rate measuring device is on your wrist every day of every week.

It's utterly incapable of measuring anything fitness related, perhaps with the exception of some cardio stuff, but it's useless for any kind of weight training.

It's baffling to me that Apple somehow got the population at large to pay for its overpriced ankle monitor and got them to upgrade to the newest model every year.

Flagged because you're not engaging curiously, and instead trying to preach your opinions and deride people based on falsehoods you believe to be true.

FD: I don't work for Apple (never have) but don't necessarily hate their devices, despite being a very ardent linux user.

The Apple watch itself has proven to be quite reliable in some independent tests; https://www.youtube.com/watch?v=qSmfM0djAE4

Gamified health is a net positive when the US and UK are dealing with an obesity crisis. Sedentary lifestyles are directly linked to this issue, and Apple even permits you to change your standing requirements (because of feedback like yours I suspect).

The battery on a new watch lasts 2 days, and after 3 years will last about a day, this depends on battery technology which degrades over time, doubly so because it's charged wirelessly and that causes stress (via excess heat) near the battery while charging. Fast charging has made this even worse, so I would consider this a knock against Apple. - But unless your battery has degraded severely: an apple watch easily lasts an entire day, even a long one.

Finally, this mindset of "people buying new ones every year" is not based in reality and I am tired of this trope. Do you buy a new Windows computer as soon as HP/Dell release one? No, because that's stupid. If Garmin or Polar release a new watch after you just bought the latest one, do you buy it? Probably not unless it has features you specifically want and are willing to sacrifice for it. It is *not* a common thing for people to upgrade a single revision of a device, wether it be apple or otherwise.

The first article I found on the topic says that most people upgrade their devices every 2-3 years, which I think is still a bit much, but at least the incremental improvements are reasonable; and in business' the lifecycle of a computer in tech companies is approx 3 years too;

https://www.phonearena.com/news/How-long-do-iPhone-users-tak...

Cellular smartwatches allow you to leave your phone behind, and still be "somewhat" connected - e.g. allowing people to reach you during emergencies, calling a taxi, checking maps etc. It can unlock my front door, be used for pay for a coffee or food, and can control all the lights in my home and allow me to answer my doorbell with a view of who is ringing it. Now it also has a digital drivers license and a mobile key to my car, so I can walk out just with my watch and nothing else, and that feels liberating.

The fitness stuff is a plus - its better than nothing, and answers simple questions like how long I've hiked or walked.

Battery life is an issue yes, but that's simple - pay more and get the $800 version ;)

> Battery life is an issue yes, but that's simple - pay more and get the $800 version ;)

If you mean Watch, the best uptime has been 2 days for me. Usually just one day. It gets much worse with cellural. I don’t buy new watch until they last for week.

Apple Watch Ultra 2 - yeah it can last 2 days if you carry it with a phone, a day without a phone and limited cellular use. It does require daily charging ideally.

This, the only "fitness tracker" I need is a sufficiently large mirror.

Self image is highly subjective and what you see in the mirror is influenced by what you ate or drank, how the lighting is positioned, your mood at that moment etc etc...

I didn't say it's perfectly accurate, but enough to see when things are getting bad enough.

Didn’t work for me until I saw pictures of myself where I didn’t recognise myself.

I saw a picture from behind and didn’t think I saw that person at my friends wedding, but then realised it was actually me, and I was not the shape I thought I was.

i hope in a positive way :)

No, I was much fatter than I appeared to myself in the mirror.

It lead to a health journey, but the gradual fattening wasn't visible until I saw myself with "fresh eyes".

Reply on news.ycombinator.com