The issue isn’t consequences for speech. Conservatives like Paul Graham are perfectly fine with consequences for certain kinds of speech (speaking out against a company in his portfolio, for instance.)
The issue under debate is what kind of speech merits what kind of consequence. The idea that there should be no consequences for speech oils only remotely believable when you are a member of the white, cis, wealthy class who’s lived without consequence for speech for their whole lives.