I'm not sure how that's different than someone sending a direct message to someone in a chat. Except now you'd definitely have no idea if someone was discussing something important and you were missing it.
With vocal interaction people expect an immediate response. With text interaction they are willing to tolerate a delay.