This argument - which I see often - ignores that land doesn’t have to be used for growing crops or grazing. Leaving some land wild allows biodiversity which has quite literally unquantifiable benefits now and far into the future. Using less land in total let’s us leave more land wild, and/or to put solar panels on it, or put it to other uses that may be beneficial.
Further, some of that land which is unsuitable for growing crops used to be - and was degraded by growing crops or grazing. Further degrading and depleting it doesn’t do us any favors.