Seems reasonable.

If someone can feed 10 people instead of 1 person, the rest has enough time at their hands to do other things.

It's rather more complicated than that. Its primary argument is that certain species of wild plants can be domesticated easier than others, and can be pushed to more productive yields through cultivation (e.g. wheat, rice), while others are more lacking (e.g. soghrum). Similarly, with animals, some regions don't have any suitable wild ones that can be domesticated as beasts of burden or food animals, others have some (e.g. pigs, alpacas), and still others scored big time (sheep, cows, horses). Once discovered, of course, these spread around, but it takes time, and is limited by climate and geographic features.

So the theory goes that Middle East and East Asia basically were the lucky ones to get that bootstrap, and other bordering regions (e.g. Europe) got it from there; while Africa, Australia and the Americas were not so lucky, and by the time they got all those things, they came on ships alongside some technologically advanced and well-armed people who weren't exactly on a humanitarian mission to spread the riches.