Fertility isn't the only factor in a family's size.
It does seem that lower income families have more children. https://www.statista.com/statistics/241530/birth-rate-by-fam...
https://www.census.gov/data/tables/time-series/demo/income-p... (The data is in an excel spreadsheet.)