Gifted programs are used to advance and maintain the status-quo of segregation in our schools. The correlation between medium household (of which Black americans are at a disadvantage) and acceptance into the gifted program is much stronger than the correlation between merit and acceptance. I've spent my life in this system and I can tell you with one hundred percent confidence that the reason many if not most of my black peers don't get into these programs is not due to any lack of merit or intelligence.
Now lets look at the second assumption, that the gifted programs provide a higher level of education. This is not true. What happens is that gifted programs get significantly higher funding than other schools and so the gifted programs can afford to give a good well rounded education, which is a good thing! The problem is that so many of our schools, especially in Brown and Black neighborhoods) are completely underfunded, without enough teachers or resources. This of course perpetuates the systemic inequities that begun this cycle in the first place thus repeating the whole cycle.
When we are calling for an end to "gifted" programs in New York, it is not because we "hate smart kids" but because we want resources to be evenly divided so that everyone can get a well funded well rounded education, not just the rich kids in gifted classes. This is not a single issue but is a larger component of an attempt to dismantle segregation in public schools. It drives me crazy to no end to see smart people fall into the ideological trap of believing gifted programs are something they aren't.