He doesn't say his parents were poor or blue-collar workers, he compares himself to a child of aristocrat.
Context is everything.
> He was a child of aristocrats; I am the child of modest workers.
When a one-digit millionaire compares himself to a billionaire like Bill Gates, using "modest" to describe one's wealth would be appropriate in this context.