You could frame the entire project of building a political system as just solving this problem without extra side effects.

If you want the right to be left alone, you have to concentrate power, so that people who don't leave other people alone can be effectively punished. That concentrated power attracts the kind of people who don't like leaving other people alone. To these power-seekers, people left alone are an opportunity cost, they could be forced to work towards some goal set by the power-seekers.