This is about silently rejecting all calls from non-contacts. You don't even know somebody tried to call, and any voicemail notification is easily lost among other e-mail and SMS and app notifications.
You're totally excluding your ability to respond to anything urgent unless you happen to check your voicemail notification at that very moment.
I'm glad your doctor calls all come from the same number -- mine don't at all. Nor do calls from delivery people asking where to safely leave a package, and so forth.
I'm glad it's not a problem for you -- I thought it wasn't a problem for me until it had negative consequences for me. So it's just to warn people, you know?