They're not open to those with learning disabilities. Or those with physical disabilities. Or emotional problems at home.
Which means they take a school's worth of finding out of the pool but leave the most expensive students behind.
So, there are clear efficiency and quality gains to be had if the system is specialized. Physical disabilities are one of the most stand-out cases, where you can invest in special access mechanisms and make special testing conditions normal if you concentrate enough people with the same needs in one place. See: Schools for the deaf, schools for the blind.
I'd agree that "specialization" is a good thing when it means that educators have the training needed to most effectively deal with other-than-typical circumstances. But it may be less of a good thing when it means segregating those students into their own classes or their own schools, which I think is what GP is saying effectively happens with charter schools. Setting aside the cost argument, giving students a chance to interact with and befriend those with different backgrounds, circumstances, and abilities benefits the students and society, though it may not be reflected in standardized test scores.