"And the hunter-gatherer tribes think the agricultural people"
The teeth, the teeth! Look at meat eater teeth compared to stone ground grain eaters teeth. Also this is a very old story which makes it unusual that they carefully avoided relative health issues such that grain eaters shrink in height and strength compared to the H-G eaters. So you've got Ms Hunter looking at a short little troll of a grain eater with rotting teeth vs studly McMeat eater... And in the other direction the ladies have little patience with the HGs because their diet gets a little sparse in the winter and they don't want to watch (more of) their kids starve.
HGs: Less of them per sq mile, taller, stronger, healthier, other than in winter when the die off, great teeth.
Farmers: More of them per sq mile, shorter, weaker, less healthy, starve less often, awful rotting teeth.
It must have been an interesting dating/social dynamic.