You need to provide them with an option to say nothing, when the conversation is over. E.g. a "[silence]" token or "[end-conversation]" token.