I've run into this very thing. The ironic thing about it is advocating for literacy, and understanding the abstract inventions of our civilization is the antithesis of elitism, because think about what would take place in a society where nobody who has the ability to influence events knows these things; where people do not know how to create their own analysis of events, to argue well, or to invent new social structures that benefit communities; to be active agents in negotiating their society's political destiny, keeping in mind the invented conceptual "guardrails" that maintain things like rule through law, and equal rights. In a society like that, elitism is all you have! Where the thought-inventions of our modern civilization are maintained, there is more possibility for creating an egalitarian society, in certain contexts, more power-sharing, and I think the evidence suggests more social progress.
This doesn't get enough credit these days, but I think the historical evidence suggests certain forms of religion have helped promote some of these civilizational inventions as virtues, on a widespread basis, supporting the secular education that spread them.
My experience has been that the reason people may put the label "elitist" on it is that learning these ideas is hard, and everybody knows that. So, there's been an understood implication that only a small number of people will be able to know most of them, and if most of the attention is devoted to inculcating these inventions in a relatively small number of people, then that will create an elite who will use their skills to the disadvantage of everyone else, who did not receive this education. What these people prefer to use instead is a lowest-common-denominator notion of egalitarianism, which doesn't promote egalitarian virtues in the long run, because it attenuates the possibility of maintaining a society where people can enact them. If we forget the hard stuff, because it's hard, we won't get the benefits they offer, over time. We will revert to a pre-modern state, because all the power to maintain our civilization is in these ideas. Keeping them as conventions/traditions is not good enough, because as we see, conventions get challenged via. a pragmatic, anti-tradition impulse with each new generation.