When "smart" parents remove their kids from the public school, the public school is worse off from having lost the student and parent would could have helped provide a better environment.
Along with the children left behind no longer getting to interact with adults who are in the more successful category, losing an entire set of role models to aspire to. How many children have benefited from their friends' parents providing excellent role models and life advice? Allowing the privileged to separate themselves into private gardens removes part of the ladder to pull yourself up by. And, along with that, it limits the mixing of the SES/racial categories, which further inhibits compassion, as you won't have grown up next to them.
A key example of the effect this category of isolation leads to is Romney's remark in 2012[1]: "... middle income is $200,000 to $250,000 and less...". Had Romney more experience (or even familiar with the vital statistics), he would never had said such an profoundly ignorant remark.
[1]https://www.yahoo.com/news/romney-middle-income-200k-250k-15...