How are you going to reach out to someone first if all communication is blocked because they don't already know you?
I think practically you'd want to be able to create time-limited, otherwise uncorrelated invite tokens/addresses that you could freely give out and deactivate later.