So if someone is talking to you, you make them remove their listening device just in case they are listening to someone else?
You’re being an asshole and you need to stop this behavior. Wait until it’s clear someone can’t hear you before making this gesture.