It’s a problem because it raises less revenue, since the tax base is narrower. It’s also more precarious, because the tax base happens to be the most mobile. To give you a sense, New York State’s tax brackets are structured similarly to the Federal government’s: the top 1% pay ~50% of the state budget. That means that if just a small handful of highly mobile people leave NY, the State loses half its revenue.

In contrast, taxing the middle class raises more money, which allows the government to spend more progressively.