Hacker News Reader

When I tried on an Apple Watch at an Apple Store, I noticed that the heart rate it displayed was definitely different from mine.

I mentioned it to the employee who was manning that table, and she told me the demo watches had some pre-programmed data for demo purposes.

I was puzzled by this since heart rate monitoring is a key feature users care about. It occurred to me that heart rate data of everyone who has tried on the watch might count as the kind of medical information you shouldn’t publish on demo watches.

Imagine ridiculous stories like “CEO of X’s heart rate was 130bpm when he tried on the new Apple Watch, says journalist who tried it on right after.”

Well, I'll be damned! Thanks for the explanation, I really appreciate it. I'm still wanting to buy one, but now I don't know what to think. I wonder how I am supposed to try one out before I buy one and to make sure it works.

I guess you would have to bring a proper heart rate monitor and compare the two readings.

Although people might give you strange looks :)

You can buy any Apple device, use if for two weeks and return if for a full refund if you don’t like it.

In a study two years ago Apple Watch was the most accurate tested device with a median error of 2%.

https://www.macrumors.com/2017/05/24/apple-watch-heart-rate-...

> It occurred to me that heart rate data of everyone who has tried on the watch might count as the kind of medical information you shouldn’t publish on demo watches.

That's an overly generous explanation. Fitbit-like devices are notoriously inaccurate[1], and the simplest explanation seems to be that their inaccuracy is unintentional.

[1] https://www.independent.co.uk/life-style/health-and-families...

The Apple Watch is in a different regulatory space than the devices listed in that source, since it's cleared as a class II medical device by the FDA. HIPAA laws will apply in some use situations, so their lawyers will have thought about it and made recommendations. (Lawyers aside, the feelings of both the medical community and Apple are generally that you shouldn't leak customer information. This seems like the kind of data that people would accidentally leave behind at the store without thinking about it, which doesn't feel right.)

https://hipaahealthlaw.foxrothschild.com/tags/class-ii-medic...

so we need to find someone with access to 2 identical watches, perhaps someone who bought some for the whole family? instead of strapping all to the same arm, just strap one on each wrist (assume the "left" heart's rate is approximately equal to "the heart on the right").

Are we claiming that every product needs 2 FDA applications: one for the end user and one for a special run of demo watches? I believe (but did not check) that demo watches would be either exempt or go unnoticed considering the amount of data it could gather...

Heart rate data gathered from a store demo device without any personally identifiable information isn't subject to HIPAA rules. If it was reading inaccurately then most likely the band was loose or the sensor was dirty.

Wrist worn optical heart rate monitors are actually very accurate, under normal circumstances when worn properly. It's easy to verify accuracy yourself by just counting your own pulse rate with your finger. Try it.

Wrist devices will give inaccurate pulse counts if the band is too loose, skin is dirty/wet, or your arm is moving around quickly. If you need accurate numbers in all circumstances then a chest strap which detects electrical signals is a much better option. I usually wear a Garmin HRM-Tri strap while exercising and it transmits a wireless signal to my fitness tracker.

Manually (as in manus, the hand) measuring your pulse while not at rest (sitting down, in a calm environment) is so error prone, it's almost useless.

The only proper comparison would be with a chest strep, which are much more accurate.

Also, it really differs per person and how you wear the watch. Just wearing it a bit differently can change the results significantly.

Sweat also gets them messed up easily.

All reasons why, if you want accurate data for workouts, you should always use a chest strep. Even if they are much more awkward.

Anecdotal, but I got an Apple Watch because I never liked the squeezing feeling of the chest strap of my Suunto. I put them to test for the first couple weeks and both at rest and while running the difference in the HR reported by both was less than 1bps.

So I ride bikes and always monitor my heart rate while working out.

I've always used a chest-based heart rate sensor until I started trying wrist-based about 4-5 years ago. I've used countless brands and models (ranging in price from about $49.99 up to about $399) on my wrist - and literally none are as accurate as the chest-based strap. The chest strap responds immediately to increases and decreases in heart rate. Whereas the wrist sensors always seem to lag, sometimes by up to a minute or so.

The wrist-based sensors always seem to differ depending on where on the wrist they are worn - high, low, inside of the wrist, outside of the wrist, etc - by maybe 5 bpm or so.

In either event, if you really want to test the accuracy of the wrist-based sensor, you should really compare it to the output of a chest-based strap.

In fact, the results will largely be personal - because the electric signals may differ per person greatly in different areas of the arm.

Needless to say, my apple watch is worn daily, but when riding, I have gone back to a chest-based strap.

That all being said, there is a guy that reviews these gadgets and will often wear 2-4 sensors while working out, and post an overlay of the results from his workouts. The baseline should generally always assume that the chest-based straps are accurate. His site is called DCRainmaker, if anyone wanted to look up his comparisons/reviews.

The wrist lag is the biggest problem I have with my Fitbit wrist band. It is OK for looking backwards of how my overall work out went, but it is useless for trying to hit a target heart rate.

I wonder if the difference in results is partially due to a difference in measurement strategy: a naive way is to wait one minute while counting the pulses which will necessarily be slower in response (it could still update the screen at a faster rate of once per second by having 60 different integers but each integer gives the count displayed on the screen when it's last second has passed, but that would still have a slow response in the sense that given a hypothetical heart rate change from a fixed constant initial rate to a new fixed higher rate, like when you got scared by say unexpected fireworks, it would still take a minute for the final rate to show up).

A faster response could be got by measuring the time between heartbeats (probably a low but even number of beats).

So, I strap all the free ones on the arm

I can't say that I know exactly how these things work, but I wouldn't be surprised if they were built to work with the blood vessels and skin thickness of a wrist, where watches are worn, rather than the skin thickness+hair+different vessel locations of a forearm.

Different parts of the body are used for different things. It's why blood oxygen monitors that work on fingers don't work on toes.

Yeah, I tried to move them about over my arm, thinking exactly the same thing as you just did. It gave me all kinds of results. Another commentor points out that the data is 'demo only' data on the in-store watches. The reason for demo only data is that the data might be considered 'medical' information and, I assume, therefore covered under HIPAA. Exposing that to just about anyone that can come in the store and see the record of the bpm is, rightly, very illegal and unethical.

The research I've read [1] says that heart rate monitor in general and Apple Watch in particular should be pretty accurate. If you have darker skin or wider wrists that accuracy may be diminished.

[1] https://www.mdpi.com/2075-4426/7/2/3

Thanks for the link!

Disclaimed: I used to work in this field, but not on the Apple Watch.

The wearing a bunch of them at the same time is going to be wrong, yeah. Wrist mounted fitness devices are calibrated to read on the top side of the wrist. Even reading on the bottom side of the wrist would be a bit different, way up the arm? Very different.

It is possible to make accurate devices that read there, they just need to be constructed to do such!

Wearables are also meant to give an accurate rating over the long term. If you go running for 30 minutes, are the overall calorie burn, average HR, max HR, and min HR, all correct? Tracking in perfect sync with the heart is nice, and of course everyone tries to do that, but if the number is off for a second here or there, well so long as the overall result is the same the usefulness hasn't really changed.

Finally, due to the way optical HR works, random unexpected, movements are really hard to deal with. It can take some time for an HR sensor to lock on to the heart rate. This isn't too long (measured in seconds, and each generation of devices is getting better and better), but it relies on regular patterned movement, such as arms pumping while running or walking, the regular motion of a bench press, etc.

Stopping and checking the HR value every few seconds to see if it is correct is literally the worst case scenario for a lot of wrist mounted sensors!

Ground truth is strapping a heart rate strap to your chest, putting on the wearable, going for a run as you normally would, and comparing the two at the end.

Multiple sources have done this, and found the Apple Watch to be best in class when compared to directly measuring HR.

tl;dr: Stopping and checking wrist mounted devices can confuse them, Apple Watch, and everyone else in this field who is serious, are tested against HR straps and they come within spitting distance of accuracy.

This is a very interesting story and I imagine there is some amount of ML used to gradually improve the accuracy of results for a specific user.

Unrelated: It is called the Apple Watch. :)

It's not an interesting story, and his method of testing was severely flawed.

There are tons of people that test these devices against chest straps online, and almost all of these bands give results that are surprisingly accurate.. or at very least about what you'd expect out of a wrist based monitor.

There's no ML. It's a direct measurement.

ML only works for these types of measurements when you have some kind of reliable source of truth or fitness function to compare against.

I have the first apple watch. (I bought the newest but haven't used it yet still boxed).

What I found was that the heart rate correlated well with taking it manually and/or a finger heart monitor (pulse ox cheap from Amazon).

But with both devices you could actually tell when it was clearly wrong independent of comparison. The point is you more or less (at least I found) could figure out when it is right and wrong just based on past use and what you would expect (feeling your pulse). So if it was off it was grossly off and obvious. That is my experience with the watch that I have had for years. Either 'correct' or 'vastly off and easy to tell that is the case'.

Some of those things stop working correctly when your skin gets damp. Think about that for a second.

An iWatch is not the pinnacle of wrist biometrics, however. I'd be a lot more concerned if a high end Garmin got it wrong, that's their whole business.

Garmin and Apple wearable devices use basically the same optical heart rate sensors. They don't make their own. Every company in the industry buys from the same couple of suppliers which have developed the technology.

Part of how you're supposed to use the dedicated higher-end models (I have a Garmin Forerunner 645) is that it's supposed to be pretty tight against your wrist. Even if you "wear your watch tight" it's probably not as tight as you're supposed to wear it for fitness purposes. If at any point you can't feel it or you forget you're wearing it, it's definitely not tight enough. This helps limit optical issues with movement but also largely eliminates the impact of sweat on the reading.

Still not nearly as accurate as the chest-based models as mentioned elsewhere, but for 95% of people it's plenty.

That's surprising, since the heart rate monitors on many phones are quite accurate. Many will actually give an ECG-like trace indicating brief changes in blood pressure. I don't see any good reason they shouldn't be extremely accurate.

Well unless you're wearing a cuff you're not getting any blood pressure readings on your phone, and any "ECG-like trace" is a doodle at best as there's no electrical monitoring. Even chest straps that measure electrical output don't give you an ECG display.

I'm not suggesting that absolute blood pressure is being measured by a phone PPG, only relative changes.

Far from being just a doodle, the PPG readout ("ECG-like trace") comes from changes in skin reflectivity that indicate relative changes in blood pressure. [1]

Obviously this isn't as good as an actual ECG (and can be disrupted by movement), but it's far more information than is needed to accurately track heart rate. These devices can be as accurate as a pulse oximeter [2], though they don't account for the movement of the user.

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4822420/ [2] https://www.ncbi.nlm.nih.gov/pubmed/28464700