Even with a large staff of park rangers and a budget for trail maintenance, trash bins, toilet facilities and informational signs about "leave no trace" ethics, it's still hard to minimize the damage tourists do in parks. And even within parks, we don't let people freely roam, we ask them to stay to trails to prevent erosion and we shut down areas seasonally to protect critical nesting, feeding or migration paths. To ask private citizens to deal with such damage to their property is unfair and unreasonable.
Further, the average city dweller has no ability to assess what land is safe for them to hike on in America. Again, for parks we have dedicated staff to put up signs and set clear rules about closures and access, but agricultural lands are far more complex. What looks like "unused" land may be grazing for cattle, flowers for bees or habit for wildlife (whether for conservation or hunting). It may have been recently sprayed with chemicals which are harmful if humans come into contact with them or it may have recently been planted and footsteps will trample early crops.
"Freedom to roam" is like suggesting that the solution to homelessness is to just let homeless people choose to sleep on anyone's couch who wasn't using it. The millions of acres of managed public lands in the US, which can have appropriate usage policies depending on environmental impact, historical significance, number of visitors, etc and which can be properly managed for the long-term balance of many diverse stakeholders needs is a far better system.