My guess is that the SHA2 of the phone number is used for exactly this feature, so you can accept connections from your contacts only (your iPhone most likely already has a SHA2 calculatedfor each contact phone number) and ignore the rest.

Your phone will probably broadcast its hash in that scenario too.