The problem is that "conspiracy theorists" and "grifters" in your mind are mainly just "people you disagree with". The problem is that not everybody agrees with that assessment, nor does that have any bearing on whether Rogan should be allowed to have such conversations.

Everyone complaining about Rogan's conversations is making the argument that they are harmful, but not a single one of these complaints is accompanied by any actual evidence of harm. Do you have any such evidence?