Hacker News Reader

Comment by bumby | Hacker News Reader

A couple things:

1) I’d like to see additional measures beyond dB. A 50dB consistent white noise does not bother me (and may even enhance my sleep) but an intermittent off/on sound of the same magnitude has a very different effect

2) what is the accuracy of the consumer metrics compared to medical benchmarks? In other words, how reliable are the sleep measures of consumer devices? Anecdotally, I’ve noticed my watch thinks I’m sleeping sometimes when I watch a movie, and I’ve heard MDs say the metrics aren’t accurate enough to make strong conclusions.

View on HN

#1 is a great idea for follow-up -- looking at variance, 90th percentile vs median, and so on. There's probably some very salient measure of sleep-affecting noise that we can derive from the data.

On the accuracy of sleep metrics, when Apple Watch makes an error, it tends to misclassify deep or REM sleep as core (light) sleep, or waking as light sleep. It's relatively rare to misclassify deep or REM sleep as awake, or deep as REM (and vice versa). That's partly why we focus on those specific sleep metrics. You can see a full confusion matrix here: https://www.empirical.health/metrics/deep-sleep-percent#accu...

Thanks! That’s great information. Do you have any data on how that performance compares to medical-grade devices?

The confusion matrix linked above compares the Apple Watch to polysomnography equipment (which is the medical device used in a sleep lab). It doesn't _quite_ answer your question, since we're assuming the polysomnography equipment is ground truth, but it at least gives an idea of the discordance between a consumer device and a medical device.

There is a field dedicated to this called as psychoacoustics. Sound pressure level (SPL) alone won’t work for this experiment like you said. There are other metrics like steady state and transient noise, masking from different frequencies having different effects on humans. It is definitely doable but far more complicated than the tweets seem to make it sound.

It would also be interesting to see the effects of intermittent noise when there is ambient background noise.

Yes i always sleep with a nature sound generator. It's comforting.

yes, that would be nice to have. but surely the data we have already would suggest for the benefit of our entire society, especially children (with the link between good sleep and mental development), we should incentivise thicker windows, better sills and other quick wins now

To the point above, we should be careful about making strong conclusions if there is a lot of uncertainty in the underlying data.

On the contrary, we gain more by making faster decisions than we do by being 100% precise

Nobody claimed we need 100% precision (or, more to the point, 100% certainty). But I would argue the level of our convictions should be proportional to the risk and level of certainty.

To the point being made, for most Americans the cost of updating your building envelope and fenestration is relatively high. Yet if we look at the accuracy of the data (compared to the medical device standard), it’s moderate accuracy. So we’re taking a high risk for something that we’re only moderately confident will work. That’s not a good tradeoff. There may be interventions that are lower risk that make for a more balanced approach.

“Move fast and break things” as a general philosophy is a risk-blind approach.

This isn't the first study on noise and sleep, in fact it's barely scientific compared to the rest of the research.

There is no scientific disagreement as to the correlation, it's just "how big" and "what specifically".

Compare the downside of "smaller impact than expected" with the cost of delay and it's a no-brainer

>smaller impact than expected

Yes, it’s about both effect size and uncertainty. That’s science. But your original post also brought policy into play. Unless you actually know that uncertainty, effect size, and those costs (including opportunity costs), you’re just making stuff up because “it just makes sense so it must be true.” That’s neither good science, nor good policy.

Advocating for big policy changes (like changing building codes) without understanding those aspects is like jumping into a pool headfirst before knowing the depth.

and this is how societal progress slowly crawls to a halt.

Moving fast and breaking things has a place. It’s just an error to think it applies to everything. There’s a reason why it’s usually relegated to relatively low-risk applications. I doubt you’d advocate an avionics engineer to take the same attitude.

You're the only one who said break things

It’s natural consequence of risk-blind policy making.

The idea that we can move fast and not break things without understanding the systemic risk is naive. I’m fine with moving fast if we understand the risk and the risk is borne by the appropriate parties. Your previous comments are a little too hand-wavy to indicate a nuanced understanding of the risk.

Reply on news.ycombinator.com