Note that the result of this was a large drop in total agricultural production. It wasn't so much that people left the cities to work the fields as that the people who lived in the cities died.

Cities have never replaced themselves, by the way - urban population has always come from people leaving the countryside. When the inflow shrinks, the city will shrink even if no one leaves.