A family member had a career in title insurance and it does matter if the neighbor puts a fence on your property. You'll have trouble getting a loan for the property, or even selling it, because while the county has the maps, and the surveyor will mark the lines, the neighbor is claiming that property with his fence. There are agricultural exceptions to this from what I remember... Also If a neighbor fences part of your property, then breaks a leg on your property behind their fence, your homeowners insurance will be responsible for the bill up to the limits in your policy.