That's a good point. But a 2.5X increase in house sizes doesn't account for the 100X rise in house prices.
If a house cost $2000 in 1950, it probably costs $200K now. It can't all be due to increase in house size.
The biggest culprits probably are tax incentives for homeownership, increased population, zoning laws limiting housing development and international investment tied to citizenship attainment.