What definition of 'often' are you using? 84% of Federal prisoners and 94% of state prisoners are in public "non-profit" prisons. [1] I don't know what percentage of probation services are conducted by private organizations, a quick search showed it's only done in 10 states, so not a majority. It also seems to be a symptom of mass incarceration - didn't really start to increase until early 90s.
1- Most corporate and income taxes aren't at the state or local level. Cities that find themselves trying to increase taxes too high find themselves in situations like my city (Baltimore) or Detroit, etc - a taxbase that flees to counties with lower rates, and the very people we're trying to help trapped in what are now sprawling cities that lack the ability to maintain their infrastructure and institutions. I would consider taxing 'correctly' an unsolved problem, especially at the local level where they have to actually convince businesses and individuals to stay in their area.
2- I'm not necessarily in favor of 'for profit' prisons but I don't believe the opponents have either a) established the connection between private prisons and increasing incarceration rates, or b) shown that states without private prisons are more willing to decrease prison populations. It's not clear to me that the voting population has required significant convincing to stay 'tough on crime'.
3- Sure and of course we continue to overburden public defenders resulting in what ought to be considered unconstitutionally deficient 'access' to legal counsel.
[1] https://www.aclu.org/issues/mass-incarceration/privatization...