Sadly, even a small, concerted group of violent actors can terrorize a society. I don't think the paramilitary core of the Nazis were a particularly large part of German Society, for example. That doesn't mean that an outright civil war would break out, but it does mean that all it takes is an organized group to upset the political balance, and disturb long respected norms.