> Those peasants could have tried their luck planting a field in the middle of nowhere, far from any village.

Except not really. Peasants were considered the property of their lords. If they decided to go and “try their luck” at a neutral location and that was reported to their lord, armed men on horseback would be very likely to show up not long after, to return them back to the lord’s land, but not without a solid beating and/or maiming first.

Sometimes, certain forms of civilization just are oppressive, rather than collaborative. They tend to be out-competed by forms that are less oppressive, true, but it’s not guaranteed and not always fast and not always permanent.