In Taubes' books, "Good Calories, Bad Calories" and "Why We Get Fat" he gives examples of populations which did not have access to fast food or supermarkets, but which nevertheless had high rates of obesity: early 20th century Pima Indians; many children in 1930s New York (pre-video games); 1970s Prague; 1960s to 1970s Jamaica.
In fact, with the latter there were many cases of obese mothers with stunted, malnourished children. How to account for that?
An early 1980s study of Mexican-Americans in Starr County, Texas found 40% of men in their thirties were obese, despite their working in agriculture or oil-fields.
Look, a lot of us middle class folks get our only physical activity from exercise. But many people do physical labor for a living. Ever seen a fat construction worker or a chubby motel maid? I have.
Physical activity has definite health benefits, but weight loss is not one of them.