There has been a decline in households with four or more people and a rise in households with one to three people, but our housing stock tends to be geared towards nuclear families with a breadwinner father, homemaker mother and 2.5 children.
Last I checked, average new homes post-2000 were more than twice the size of new homes in the 1950s and held one less person on average. So you are seeing couples with one child buy four bedroom homes and turn the extra bedrooms into an office and a playroom or hobby space.
Meanwhile, the poorest of the poor are often unable to find housing at all. We have a lot of homeless people.