To really get to the truths behind this opinion, it is important to distinguish one's political goals from the politically implemented policies in pursuit of those goals.
I fully agree with you that the average person is not well equipped to decide on policies. However, the vast majority of people in the elite aren't equipped for that either! Look at somebody like Bill Gross, for example, who was removed from PIMCO after continuously making bad investments based on his misunderstanding of basic economic facts. I wouldn't want somebody like that calling the shots in government.
Honestly, I would expect the connection between wealth and quality of policy analysis to be relatively weak. Education is certainly a confounding variable, but not all education is equal. (A good computer scientist doesn't necessarily make a good economist.) In fact, in certain policy areas such as economics, being part of the elite - which usually means being a high-level executive - is probably a negative for the quality of your policy analysis, because executives see the world from a micro-economic perspective rather than from the macro perspective that is needed in politics.
In any case - I agree that a certain level of "elitism" is probably justified when it comes to the question of policy analysis, though you really have to be very careful who you think the "elite" is in that context.
However, if you leave politics up to an elite, then the goals of politics will also shift to match the goals of that elite, which is probably not a good representation of the goals of the overall population. And therein lies the real crux, because when it comes to deciding what the political goals should be, everybody really does have the same qualification. And this is ultimately the reason why democracy is the right way to go.
The tough problem is to establish institutions that translate democratically established goals into good policy.