The nutrient-deficient and unvaried diet, poor hygiene/concentration of people, and difficult menial workload from farming gave us very high infant mortality rate, higher disease rate, shrinking height and all sorts of health problems, repetitive strain leading to bodies just completely worn out by age 50, etc.
As people started eating better etc. after getting away from a peasant lifestyle, average height increased by 30cm, infant mortality went way down, life expectancy and quality of life for elderly people went way up, etc.
Our bodies to some extent started reverting to their “natural” shape, but of course being bottle-fed, wearing rigid shoes, sitting inside all day staring at close work from birth onward, a diet with increasing amounts of refined sugar/starch and plenty of alcohol etc. has certainly had its physical effects, and is leaving us with small jaws, dental caries, warped feet, inflexible ankles (unable to squat e.g.), decreasing levels of physical strength, back problems, diabetes, dementia in old age, ....
True, but I think the differences can still be explained 99% by changes in lifestyle. Genetically we are still overwhelmingly similar - with a few tiny differences like lactose tolerance, etc.