Less animals stand to die at the blades of the harvester when one chooses to stop eating animals altogether. You can't eliminate suffering, but you can take active measures to reduce it.
The “less violence” argument doesn’t hold sway with some. That’s a purely subjective assessment that not everyone agrees with. For example, I don’t consider the slaughter of animals for food to be particularly more violent than the wholesale shredding of plant fibers at harvest.
You probably think of "violence" differently than most people then.