Blowing up buildings, murdering jews, plotting assassinations etc. is substantially damaging. It is "extremist".
The FBI has considered right wing extremism a serious threat to the US since before Ruby Ridge and Waco. The turner diaries was written before either.
And the KKK was exactly a militant right wing political group, with members being politicians, sometimes openly.