The amount of farming land destroyed for plants is far worse than grazing land however.
looking at your post in another way, the reason more land is used for plants is because of feed for the cattle such as soy and corn, not for human consumption.
Beef, on average, requires 16726m^3 per ton of product, whereas wheat requires 1437m^3/ton
http://waterfootprint.org/media/downloads/Report12.pdf
Not to mention that to successfully raise cattle year-round, you need to rotate feed crops for them as well. So you require not only land and resources for the cattle themselves, but also for their feed.
Meat and wheat serve different nutritional roles.
What do you mean by "destroyed" and "grazing land"?
The only way I can understand your comment to make sense is to assume that you believe corn, soy, and wheat fields to be destroyed by farming activities, but grazing land to not be destroyed.
It's true that crop fields are often devoid of ecological diversity, require significant artificial fertilization and watering, and do contribute to topsoil loss. And some cows graze on huge ranches with little human management, walking through federal land in the open West among the sagebrush, evoking images of ancient herds of buffalo and a pristine ecological paradise: simply replace the buffalo with beef cattle, and skim a little off the top of that ecological system, rather than plowing it all under and growing row crops, right? Wrong.
In reality, the vast majority of cows are, if not raised entirely, finished on feed lots: Small plots of mud and dust that cannot support a tenth of the plant life which would be required to grow the cattle that tread on them. The cows are fed grains like corn, hay, and alfalfa, soybean meal, and protein concentrates, which are grown off-site. A human with a tractor and fertilizer is able to get a lot more calories per acre with which to grow cattle than a cow grazing in and 'managing' its own field of grass with all the agricultural knowledge in its cow brain and the limited tools - its mouth and hooves - available to it. More at [1].
Furthermore, a cattle grazing operation - insofar is far from an ecological paradise. The cows destroy natural habitats, and human breeding of this herd and culling of its outputs is far, far greater than what a natural ecosystem would produce. More at [2].
[1]: https://www.theatlantic.com/health/archive/2011/12/feedlots-...
[2]: http://www.biologicaldiversity.org/programs/public_lands/gra...
However, there are some factors that need to be taken into account. Farmland and crops that can be used for grazing are not necessarily capable of producing anything fit for human consumption. Livestock can eat cellulose-heavy plants (grasses and shrubs and so on) and convert them to meat and milk, which we can then eat.
The biggest problem then turns out to be cattle. We are eating way too much of it, it's an ineffective process and needlessly resource intensive. We should obviously grow food crops fit for humans where it's possible, but we should also convert to primarily chickens, pigs and other animals that are hardier than cattle and significantly less resource intensive to keep.
Beef has an undeserved reputation for being fancy and higher class than other meats. Add this to the deleterious health effects of a diet high in red meat (generally specified as meat from ruminant animals), and I've personally cut beef almost entirely out of my diet. I prefer chicken, anyway.