But often the land belonged to the lords or at least they could take it within their region at their pleasing. There were not that much "neutral locations". That wilderness was common land or hunting grounds belonging to someone.
But it's not exactly irrelevant, either, right? It does add to the confusion to leave it out, so, my bad.
As far as the black plague, what gave even serfs so much leverage was the fact that there were so few people in all categories around anymore. Fewer peasants to work the land, so the prices of the free peasants went up commensurately. But even serfs who wants to run far away and set themselves up as free peasants elsewhere could frequently get away with it because a) fewer men on horseback to spare to chase them down, and b) the other lord they went and rented land from didn't have enough peasants to work his land either, and had a real vested interest in making sure they weren't caught and brought back to their original lord.