Simply imagine that the user thought that, by clicking the "X" button, he closed the app and got off the vocal channel so that now nobody can hear him. Now a close relative ask him some other personnal info that he don't want to share (Health info, credit card etc..), and other people in the vocal channel hear it. Well, the user has been harmed.
So yeah, not about "disliking the UI" but about protecting you, the user