We might have to. Fringe groups like neo-nazis thrive on suppression. Censoring them may serve to validate their narrative of fighting for a cause despite oppression. It also shows them people are taking them seriously.
The YouTube recommendation engine is a far bigger culprit. It doesn't take many clicks to go from fairly innocuous, but kid-popular stuff like H3H3 or PewDiePie to one anti-sjw or cultural war video and then the recommendation engine will hand hold them right down the right wing extremism tunnel (or continually show up until they do)