The ugly truth is that at the root of it all, violence is the foundation upon which society sits. Society is held together by the threat of violence.
A pacifist gets to live their pacifist lives, because they are kept safe by those who would inflict violence upon those who would threaten society.