Unless they're actually offended by non-violence, in which case they probably did you a favor.
Most would not say they are offended by non-violence, and most would actually advocate non-violence. The problem comes in when you start pointing out that logical and consistent definitions of violence/aggression preclude a lot of the things people value and think noble. To some, mentioning you are Libertarian inherently means you are saying all of their "noble" and "good" ideas should be scrapped because they rely on aggression. Hence the cliched "you want poor people to starve" criticism of Libertarian-like beliefs.
Violence is also a bit of a misleading word, in that it paints a very physical picture. Threat of violence can be seen as aggression, and is on-par with actual violence because the implication is that if compliance is not achieved, the aggressor will exercise violence. Had to clarify a bit, as I did switch the discussion from violence to aggression, which is what libertarianism and anarcho-capitalism is mainly about.
Their values can simply be different, and it is valid for them to disagree with you based on the conclusions they get from those values.
Aggression is defined differently by different groups after all, so the set of actions that a libertarian objects to based on their understanding of "aggression" would be different to a set of actions that someone else would object to based on a different understanding of "aggression".
Obviously you disagree with their conclusion of dismissing libertarianism and blocking libertarians, and that's fine. What's less fine - in my eyes - is to imply that the disagreement comes from the fact that you are the only one who understands some basic Truth about the world and those that disagree are foolish and cliche for disagreeing with you.
That's the sort of thing that I associate with religion, rather than an honest intellectual discussion of political merits of an ideology.
But to your point - yes of course it's ok to disassociate with someone who is advocating violence or National Socialism, because it can reasonably be inferred that said person isn't rational, or has a twisted sense of morality than makes associating with them damaging to yourself. However, I'm assuming that's not the case here - it had come in vogue in SWJ and intellectual circles to immediately disregard anyone who has an incorrect opinion. Vote republican? You obviously are a terrible human being. Fall in line with the Democrats except you think abortion is wrong? Kill yourself.
Given the very serious discussions in US politics about placing US citizens in concentration camps and refusing them re-entry to the country based on their religion, the question about at what point a party becomes the National Socialists must be a legitimate one to ask. Clearly we're not there yet. But also, clearly there's a nonzero chance that we might find ourselves there very soon.
If you want extremist views to have free reign, then keep avoid them until they form a broad enough coalition of the ignored and disaffected to take power, I guess.