It’s a major vulnerability, but I think a valid solution would be for there to be an organization that wears uniforms and has a strict no-violence policy to perform peaceful protests.
That way when agitators show up they can be seen as visually different and distinct.