"all of the other people come together to ask you to stop, thus infringing on your liberties."
You've very insight-fully described the policy debate/disagreement between people in pretty much any liberal democracy. Respecting my property rights is actually an infringement on your liberty. Not being able to legally kill me without some just cause is also an infringement on your liberty. What's often lost in the conversation on liberty is that everyone agrees that its a matter of degree. IE -- everyone having absolute liberty isn't viable. Absolute liberty would effectively just be an anarchy. Its just a matter of where the lines are. But somehow the conversation ends up getting reduced absolutes on both sides.