The yard analogy doesn't work. You can't stand in a certain place in your yard when someone else is standing there. They are denying you the use of your property, however temporarily. The same goes for people standing in your house (which brings up privacy problems that are orthogonal to intellectual property). There are also maintenance costs - your yard might be ruined by hundreds of people walking over it, and you'd have to pay for its upkeep.

Those are both good reasons for protecting physical private property that don't apply to ideas.