That's... how you got places.
If you actually damaged something -- broke a fence, stole a horse, dug a ditch, whatever -- then, yeah, you'd get in trouble if you got caught.
I heard from a friend that moved out to a rural area that especially the newcomers moving in from cities are super-sensitive about "their property". His neighbor -- from NYC -- threatened a lawsuit because my buddy's truck broke off some branches that extended out into the common road.
Folks that have been living out there awhile are cautious with outsiders, but overall a lot more neighborly, just because you have to be.