The usual counterargument to this specific line of reasoning is outlined in [0]. TL;DR vastly more animals die incidentally in the process of growing feed for livestock than in the process of growing plants for human consumption. The math is very simple; livestock outnumber us dozens to one, and consume comparable amounts of food on average (cattle really drag the mean here.)
More generally, you'll always be able to whatabout vegans, because to live is to kill --- it's completely impossible to live in modern society and not harm something somehow. I feel that I'm obligated to do the best I can under the circumstances.
[0] https://www.surgeactivism.org/articles/debunked-do-vegans-ki...