This is very true. However I still think among people I knew, wealth counts for more.
For example when I was in grad school one of the professors in our department liked to proudly remark that he was proof that you could have a viable career even if you change your field substantially in a second post-doc. Eventually I heard someone else explain it as, "Well, if your parents make a large donation to the lab then the PI will accept you as a free (no cost to the lab) post-doc and train you." In contrast, people I knew from academic families but with relatively middle-class financial means still got railroaded / pigeonholed into the areas where they specialized, even though they were much better at navigating academia than I was.
I know, the above is just anecdotes. How could we resolve it? I dunno, run a study? But as you say wealth is much easier to measure than culture, so we may be stuck with our respective perspectives for a while...