Being intolerant of intolerance is the only way to foster a tolerant society. It's the solution to the Paradox of Tolerance.
If you allow intolerant views, you normalise intolerance. If you normalise intolerance, you ALSO normalise intolerant acts, and they WILL follow naturally.
If you normalize cracking down on views on the basis that they're intolerant, the people who make that determination end up with a lot of unchecked power. In France, for example, you can be jailed for wearing a t-shirt that says "boycott Israel" today - it's hate speech. In Germany, many flags used in Rojava are banned as extremist symbols. And note that despite all these laws, RN and AfD still exist and thrive.
If you allow intolerance to spread, it WILL, without fail, hurt people. Sometime, immense number of people. This is not theoretical. It happens, again and again, and the only way to prevent it is to oppose intolerance at every stage.
Your abstract ideology of absolutely free speech is not in any way worth the immense pain and suffering that will, inevitably, follow.
On the other hand, policing actions is much easier, because the more consequential ones are also the more prominent - you don't need a surveillance state to deal with them, you just need a reaction force.
You most certainly can draw a line between actions and views, and fortunately most laws do.
Don't be friends with a racist. Make sure they understand their views are not acceptable. Do your part today.
In my experience, people are seldom racist for the sake of being hateful. Rather than applying an arbitrary label to someone so that you can lazily dismiss them as evil (which is just as bigoted as the bigotry you claim to oppose), it's better to find the underlying cause of the apparent racism, and addressing that concretely instead.