I think you are drastically oversimplifying the carbon impact of modern agriculture. Artificial fertilizer production, irrigation, mechanized farm equipment all substantially contribute to CO2 emissions and other serious environmental impacts.
Edit: Reading the EPA report, the sources I mentioned aren't even in the top 3: https://www.epa.gov/ghgemissions/sources-greenhouse-gas-emis....
Soil management practices (aka tilling) causing N2O release, digestive methane from cattle, methane, and methane and N2O from manure handling. Fertilizer was lumped in with crop residue burning and rice cultivation in 4th place.
Two of the top three are directly related to animal husbandry, and the rest are heavily impacted by the vast amount of land used to grow feed for cattle in feedlots instead of directly feeding people.