the crux of ag soil "energy" is just nitrates and stuff. some plants like beans and lentils recharge nitrogen in the soil (hence the old native american trinity or whatever, corn, beans, and some other plant). fertilizer, that's just nitrates. poop and pee (natural fertilizer).. yup, nitrates.
Basic reason would tell us, unless we're using a nitrogen-fixing plant like beans, unless we're using fertilizer, the plant's gonna suck up nitrates from the soil, get harvested, and now there's less nitrates in the soil. Weakening the soil quality thusly. Pretty straightforward.
Biologically you could then go on to argue micronutrients along the same lines, with poop and pee more likely to contain those things, since aminals also are eurkaryotes so probably depend on some of the same