Yeah that's where everything gets complicated, obviously people's natural rights can conflict and that's where you need to set up some sort of rules based system, but the point is that you don't infringe on someone's natural rights unless there's a direct conflict with someone else's natural rights. For example your natural rights to bear a weapon cause a conflict if you are bearing them in someone's house in order to rob them after you just broke in. Conflicts with their right to have property, etc.