Source? It would surprise me to see that many people voluntarily give up their farmland rather than, say, selling it to industrial farmers or real estate developers.
Of course, that is anecdotal, you can read more here: https://www.bostonglobe.com/metro/2013/08/31/new-england-see...
In my area, once you get passed a point south of me, those walls are fewer and fewer. There are areas where it was never farmed and old growth trees still exist.
What is great is you can walk out to places and stand there and be reasonably certain that you're the first human to have ever stood in that spot.
edit: removed link to an article that has been posted several times.
Once you get somewhere more rural/remote, the land is going to have even less value.
Or go walking through the forest, and count how many stone walls you have to clamber over. Every one used to mark the edges of a field or a pasture that was hacked out of the forest, and then was allowed to grow back up when farming the cold, rocky soil of New England was no longer cost-effective.