Oh yeah, the "burglars know when you're not at home" scenario... I feel like maybe there's something else, but this one's definitely a standard IoT security checklist item.
Privacy breach happens, and employers, potential employers, banks, insurance investigators etc. suddenly have access to that information by name. Mom gets fired. Dad will never teach again. Dad can't get a life insurance because of his current health status.
Some of these scenarios are unlikely, and some of them might seem like net benefits to society if we squint a little. But the breach of privacy was detrimental to people, and we all do things that we don't want everyone or anyone else to know about.