We've actually had many conversations surrounding this exact thing. Currently users can join any open conversation but we have plans to add some very simple privacy options so that some conversations won't be intruded on.