Perhaps this is more of the case in the UK where class is more of a salient aspect of life than in the US?
Of course class exists in the US, but if you're able to move up in a tax bracket people won't judge you for having come from a lower tax bracket.
Rubbing elbows with rich investors and CEOs from affluent backgrounds is definitely more difficult for someone from a lower class background.
With maths, you can work as a professor or work in finance. Two domains with limited demands that are highly competitive.