I completely agree with you. That's exactly what would happen, people would vote with their feet against dictators and abuse.

Borders are either to stop people from leaving or to stop people from arriving. In the first case it is dictators creating a captive audience they can fleece and manipulate. In the second it's the haves that desperately want to avoid being confronted with haves-not.

Take away the borders and you'll reach a natural state of equilibrium in a very short time. A lot of holy apple-carts will be upset by such a move.