I was always of the opinion that separating based on ability level was not about improving the baseline, but about allowing those able to learn greater depth to do so.
Not every student is going to earn a Fields Medal or develop new medical procedures, and frankly, that's ok.
But we do need some who will push the boundaries.
The only issue I have is that the selection process itself has not been entirely merit-based, frequently showing preference for students from certain backgrounds at the expense of gifted children of color or from less wealthy backgrounds (and at the expense of society as a whole, who has lost out on the benefits those children could have provided, given the opportunity).