Bringing home some grains of that wild wheat that grows just a tiny bit larger. Befriending a wolf that is a little less distrustful of you. Picking that apple with the slightly smaller seeds. And so on.
Without those, you really are better off remaining a hunter-gatherer. People became significantly shorter when they first adopted a sedentary lifestyle.