I don't think I made that claim? But feel free to quote me if you think I did.
My claim is that we all depend on each other and that if you dropped a gifted person in the woods and isolated them they wouldn't achieve much useful. They might not even survive. It's this interdependencey that gets ignored when we idolize one of the societal roles too much. People smart at math are important. But so are the people who give them the free time to do that by building highways, serving them food, or making the computer they use. In a market economy those folks are valued less and I'm not here to debate economic systems and the morality of that. Just that if we do not make an initial investment in those folks, the gifted folks won't have as much time to make their contributions to society. So there is actually an important balance to seek when trying to optimize for the greatest benefit to society. Also keep in mind that societies have executed the elite when things get too unbalanced.
I already mentioned in another comment that the ideal would be that everyone gets the education investment that challenges them to learn as much as they can. Since we aren't there yet I don't think it's unfair to give an average and basic education and to let parents of gifted children take on some of the educational responsibilities. The state can only do so much.
As far as your question about the single dollar spent on the non-gifted kid but not spent on the gifted kid, I'm not aware of that happening. Don't all children in the same area have access to the same public education?
As far as metrics or sources, no I don't have any. My argument is mostly persuasive, but I hope based on basic things we can all observe or repeated chapters from human history. However that doesn't exclude me from curiously asking for sources for claims from others. Sources are an opportunity for me to learn more than fits in an HN comment.