You are completely glossing over the fact that animals need to be fed.

Sustainable grazing (most grazing isn't) is a perfectly good use of marginal land, but that cannot produce anywhere near the meat we consume. Most animal production requires producing feed from other agricultural products. It is inherently less efficient to do this. Chicken is more efficient than beef, but still much worse than just eating plants directly.