That’s like saying that property rights are the “freedom to deny others’ freedom.” Banding together and controlling territory is one of the most basic functions of human societies. There is no “right” to go onto territory another group has developed, cultivated, and controls.