Accuracy of Commercial Sleep-Trackers Compared to Research-Grade Tools
mdpi.com
mdpi.com
I was a hardcore flu.x/nightshift user. I left most lights off once the sun set. Now I'm a lot less convinced that lighting stuff matters for sleeping (but I would still follow it given the the whole cancer-light link thing.)
Of course, all of the common sense stuff still applies: staying up all night, sleeping erratic schedules, drinking a gallon of water before going to bed, drinking a bunch of caffeine late at night, medications, stimulants, cocaine, trying to sleep in a room that's 90F+, having sleep apnea, someone drilling a jackhammer outside your bedroom window etc. If you do a combination of those things there is probably zero you can do to sleep consistently and well.
I really wish I had understood this decades ago. Getting older it hits me how trivially simple solutions to certain problems are despite most running around in circles trying to solve them the entirety of their lives. To be fair, for sleeping poorly, I got an incredible amount of work done sitting in front of a PC 12 hours a day. That's a number you probably can't hit if you are getting adequate exercise, quality food, and consistent deep sleep.
Oh, but you can't (can?) sell a self help book that way. It would only be a few pages long and everyone in the media would shit all over the author if it ever got traction. Hordes of bots on Reddit would be having fake conversations in threads that cry "science!" in an attempt to protect book sales based on TED talks and trending morning show style topics.
Same here. Plus drinking much less, I rarely drink these days, and when I do it’s not more than 2 beers or glasses of wine.
But mostly it’s the exercise. If I didn’t go for a nice long walk, at minimum, then I know I’m gonna sleep like crap.
I have found that a cool or cold shower before bed really helps. If I am warmed up from exercise (or just sitting in a warm room) I find it much harder to go to sleep. But not too cold for too long - that can be stimulating; not what I want!
> There are limitations to consider in the interpretation of these results. First, devices were worn for one night.
That is odd.
What I noticed is that my Garmin watch may be wrong at times, but overall provides a fairly good and very useful assessment, especially comparatively day vs day and overall for trends.
In addition it takes a bunch of time for the watch to work out enough data to tune the algorithms for a particular person: some UI bits explicitly call out that for accurate results the watch should be worn continuously for ~2 weeks.
I've now been wearing this particular one (Fenix 7) for 6 months, a serious upgrade from a Forerunner 735xt continuously worn for 5 years. 95% of the time the watch is stupidly accurate in ways it has no business being as a simple wrist-worn set of sensors.
The metrics aren't really actionable either. I got a bad night of sleep, so what? I know it. I have low HRV because I drank alcohol/over-trained/am ill. I don’t need a watch to tell me that. I know it.
It’s just collecting data for the sake of collecting data.
No, sleep is one of the most important parameters for health!
If in 2024 you are not collecting data on your sleep to take action (and for ex, stop overtraining, reduce drinking etc) to reduce the risks of future brain problems, you are doing it wrong.
> Even if it was accurate, I can just tell by feel
I can't, and I fear it may be the same "overconfidence" issue that cause car accidents when drunk.
I collect logs with my sqlite based bash history collection that I've been running for 5 years: https://github.com/csdvrx/bash-timestamping-sqlite
Thanks to my garmin, I have identified a pattern where I feel ok, but my garmin reported less REM sleep than usual: in the day, I run more commands than usual but I have way more mistakes (as can be seen by the non-zero return values)
So now, when my watch reports low REM sleep, I try to be very careful to avoid negative-productivity days
If the software behaves no differently from one night to the next, that is, if you can take any random night's assessment and it would be as likely to be accurate as the first, then it does not matter that they had participants only wear it one night. You'd get more data (from multiple nights) without needing to recruit more participants, but you'd also be biasing towards whether the tracker works well for this set of people (probably a worthwhile trade-off, up to a point)
Your watch says to use it for a while. It could be that that's just because it's not flawless and they want people to see it's often correct even if they got unlucky the first night(s). Or maybe your watch works differently from the ones used in this study
I'm also surprised they only did one night, both because you get a lot more data per amount of effort by having participants keep it for a month, and because I'd not be surprised if the software is more comparative than absolute. But I don't know if we can draw real conclusions from that
In other words: these devices are all terrible, and shouldn't be trusted. Pretty different from the final conclusion, which is probably "soft" to get additional grants from the commercial entities selling these devices.
Detecting light versus deep sleep accurately with wristwatches or devices under the mattresses might not be possible. Especially not when accuracy is supposed to be comparable to multiple electrodes places directly on the head to measure brainwaves.
edit: Also, the last time I did a sleep study at home it had rather little correlation to how I normally sleep except in terms of apnea. Having wires, electrodes, a chest strap and a finger O2 monitor strapped to me did not lead to a pleasant night of sleep.
Doesn't that assume that the errors will be consistent and precise enough that any trends don't get totally wiped out?
I own both a Fitbit Vivo Active 4S and a Garmin VivoSmart 4.
In the daytime, as a watch, I much prefer the Garmin, but at night, as a sleep tracker, which is the main thing I want from these watches, I only trust the data from the Fitbit.
If I'm concerned about actual total sleep times, I do generally need to edit the Fitbit data in the morning. Over the years, my Fitbit watches have fairly consistently confused idle time sitting on a couch watching videos or lying in bed reading on my phone before or after sleeping as sleep. The data edit tools in the app fortunately make it easy to correct this (I'm not someone who falls asleep with the video playing, so I generally know when I turned off the activity at night and when I woke for the morning).
The data during the night, which is what I care about, I've come to trust on Fitbit watches over the years. I can't know if it's precisely accurate, but it's definitely directionally correct (if there's an overall scale factor error or similar from true sleep quality and time numbers I don't really care as the measurements are consistent enough that I can use them and benefit from them).
My Garmin I view as a random number generator as far as sleep tracking. My perception is it's probably a bit better about not misclassifying idle time before and after sleep, but not accurate enough to be actually correct, and I have not been able to find any simple data editing feature in the app to fix the misclassifications, so the wrong data stays wrong.
The Garmin data during the night I have zero confidence in, as anecdotally it feels far less "believable" than what I see reported by my Fitbit.
Edit added: another commenter just mentioned that Garmin uses a learning algorithm that needs about two weeks of data to hit good data quality. I've done a couple periods of sleep tracking with my Garmin, but I'm guessing I've abandoned it due to poor data quality each time at around two weeks. Sounds like I should give it another try and force myself to wait a month or so before making any assessments.
As I am an avid runner and I train daily, I pay close attention to all sleep, heart rate, stress, and running performance numbers.
Unfortunately, it will frequently say I had good night of sleep when I definitely feel groggy or the opposite.
As to misclassifying idle time before and after sleep, I recognise that simple lifestyle adjustment could fix it (just don't browse on your phone before sleep!) But that of course won't fix it if you try to lie still but can't fall asleep.
Regarding grogginess or lack thereof, what I noticed is that sometimes I feel that way even though the watch says I had good rest, but it's only so at the beginning of the day; afterwards I'm okay.
Conversely I may have had a bad night and still feel okay, but what I noticed is that if I don't pay attention to the watch I'm actually plowing through and don't notice I'm wearing myself out, which I then have to repay the following days.
Also the watch gives a) objective measurements (whereas my feeling is subjective) and b) measures only a bunch of things but not everything (e.g I noticed eating pasta or something weighting on my mind and morale can both make me experience brain fog that the watch does not account for)
For some reason we had to slap complicated sleep scores and sleep classification on these things to take them seriously when all you needed was simple data gathered consistently over time.
Haven’t used it in years.
Do you have any recommendations for how to find a dependable overnight pulse oximiter on alibaba? I’ve never shopped there.
1 https://sleepyhead.software.informer.com/
Edit: apparently sleepyhead was shutdown and replaced by Oscar. https://www.apneaboard.com/sleepyhead/
Apple Watch: Scientific Sleep Lab Test (Ultra vs 8) (2023) - https://www.youtube.com/watch?v=BQMIAkt4P48
Apple Watch: Scientific Sleep Test (2022) - https://www.youtube.com/watch?v=LPqtfC70QTU
(Thanks to @LikelyClueless for the channel recommendation!)
- WatchOS 9 beta introduces sleep stage tracking to the Apple Watch using a machine learning algorithm trained on polysomnography and Apple Watch heart rate and movement data.
- The author tested the Apple Watch's sleep tracking against an EEG headband over 18 nights and found it agreed very well, especially for deep sleep tracking.
- The Apple Watch detected 89% of deep sleep, 84% of light sleep, and 67% of REM sleep correctly compared to the EEG device.
- Overall sleep stage tracking accuracy was better than the 37 other devices the author has tested.
- The Apple Watch also detected sleep cycles and awake periods reasonably well compared to the EEG.
- Falling asleep and waking up times mostly differed by less than 10 minutes from the EEG.
- Future improvements could include sleep coaching based on metrics.
- Results may vary for other populations not well represented in the algorithm's training data.
- The author used a beta version and polysomnography would provide better validation.
- The Apple Watch's performance makes it a current leader for sleep and heart rate tracking accuracy.
What I do feel I need is an effective way (including cost-wise) to mitigate thermoregulation issues at night, i.e. to avoid waking up hot or cold, dialing in the correct bedding temperature at any moment.
I thought a wool comforter might be the answer given its reputation, but I still fuss around with adding or removing layers. I basically sleep with a stack that changes depending on the weather.
My point is, I think these types of devices are wonderful screening tools. Cheap and imagine all the prevention we could have by screening widely and early.
We work in the sleeptech/neurotech space, and there is so much crap in this space. We're want to ensure we're taken seriously, so I'd be keen to hear your thoughts on why you think Eight Sleep isn't presented as trustworthy (if that is what you're saying).
I think they actually have the best reputation, and have taken over from Chili.
We're not in the heating/cooling space, so non-competitive.
If it was $2 a month then maybe and I'm a guy that loathes subscription costs.
I'm looking into this sort of thing as i'm hopeful it'll help my wife with thyroid issues get better sleep. but eightsleep is right off the table with those subs prices
Testing them for 1 night without having worn them for a month or two first is silly.
I personally believe that the consumer grade devices are "good enough" because the micro-data is not that valuable anyway, and the way PSG works, there is no "correct" score. Get 5 different polysomnography's, you'll get 5 different measures of the same sleep.
Once you're doing the sleep hygiene stuff, and experimented with all the magnesium stacks, ashwaganda, CBD, etc, etc, what do you do to actually ensure you are doing the most for your sleep?
This is why I got into the sleeptech/neurotech space, and for the last few years we've been building a headband that monitors brain activity and increases the efficiency of deep sleep. The tech is backed by peer-reviewed and published research from discoveries which began in 2013. https://soundmind.co/research
Yes, people like data and want to "understand" our sleep, but the quality of sleep is more than just the number of minutes you spend in each stage of sleep, just like fitness is more than the number of minutes you spend in each cardiovascular training stage.
Our understanding of sleep today is at the same place our understanding of fitness was in the 70s, and sleep is just beginning to get is due importance. What we'll learn in the next 20 years, I believe, will lead to significant changes in health and longevity.
Of course, thanks to capitalism, the device is marketed as disposable (I love how this is somehow portrayed as a bonus in the marketing materials). You're literally supposed to throw it away after a single use.
So questions:
1. I've searched online but haven't found anything about people successfully hacking the device so it can be used multiple times. Has anyone seen any succesful hacks of this? If not, can anyone point me to some resources where I would even start to try to hack it? It communicates with the app over bluetooth.
2. Does anyone know of other medical-grade sensors that are usable by consumers that are not disposable?
I ask this because in the past I've just used the "Sleep as Android" app with a smartwatch and it was horribly wrong. It just checks for movement, and there are lots of times when I an exhausted, staring up at my ceiling not getting any sleep, but I'm not moving because I'm stuck in this exhaustion-but-not-sleeping state.