A person or group of people that I've met personally and don't change often is much different than letting a random low-paid employee/contractor into your house.

I only allow people I trust to enter my house when I'm not home.

Not to mention the actual security of locks. People are discussing this functionality as if standard doors and locks are strong enough to keep intruders out. No, they keep honest people out; a real intruder can do things like open or break a window, pick a lock (or use a bump key), or kick in the door. Those are all a lot easier than hacking your fancy IoT locks and jamming Wi-Fi.

Never expected I'd hear "letting strangers into your house" be described as an "odd but widespread phobia".

For a certain class of people I'm sure that's true but I would bet 80%+ do not have people coming into their homes providing services when no one is home.