I think hate speech is a bit like drink driving. Perfectly harmless itself but can cause follow-on real world problems that do impact people's lives - ie discrimination in hiring/etc.