We like to think of education as an equalizing part of our society. No matter how poor you are, you can go to K-12 (or nearly all those grades) and get educated. Likewise, we generally try to accommodate people regardless of income at the third level - especially at elite institutions whose endowments can carry them. However, education isn't socially equalizing.
Smart people generally learn easier and an hour of teaching them creates a bigger difference than an hour of teaching someone less smart. So, if we allocate the same resources to the smart and the less smart, we're creating inequality. In effect, we're taking people who had the most inherent advantages and giving them even more advantages. This gets quite pronounced at the third level. While different school districts have different resources K-12, our elite universities have vastly more resources than our mid-range ones. Not only that, but they become a network for those who are already advantaged to meet the other advantaged people.*
Is a smart person slighted by getting less attention in school? I would argue no. I personally like living in a society that's a little more equal - one that helps those with fewer inherent gifts to achieve more while also helping those with great gifts, if just a little less. I also acknowledge that there are benefits to me of helping those who have a little more trouble achieving to become more educated.
Yes, it can be annoying when one sees those who don't put in the work, don't learn as easily, or act out being given extra attention/help/resources. However, this isn't that different from our tax system. The Republican argument against taxing the rich more is that they created that income and should get to keep it. However, some of what we achieve isn't our own. We have gifts that we have received from birth that are unequal. Likewise, we see the "lazy" and "trouble-making" students in much the same light that people on public benefits are painted by politicians. Society is taking resources from us and spending it on them?!? How outrageous!
So, do I feel like we have gotten the short shrift? Sometimes (I'm only human), but I try to remember that I've been blessed with so many advantages (being born into a rich, western economy, having a high ability to learn things, having the time and resources to study things, etc.). Would you trade your intelligence or ability to concentrate for the additional resources offered to those with less? If you were to sit down today and design how the educational system should allocate resources knowing that your intelligence would be randomly re-assigned afterward, would you allocate more resources to the smart (knowing you might not be smart afterward)? I'm not posing the question as a hypothetical with a clear answer - I genuinely believe that people have to explore this in their own conscience.
I don't like paying taxes either, but I see a certain fairness in it - I've been blessed with gifts that are not entirely my creation. But it's completely reasonable to find it incredibly annoying - those are resources that could be used to help smart adolescents achieve better things. Still, I'm not sure it would create a society that I'd rather live in.
*In this case, "advantaged people" doesn't mean economically advantaged, but intellectually advantaged. For example, Harvard doesn't generally care if you came from money, but isn't generally taking people who don't have a lot of intellectual advantages.