I would speculate that it is lack of sufficient competition in the market. Therefore the cost of living for single workers in single worker dwellings raises to eliminate any surplus.
Most larger dwellings are similarly set by the price that can be afforded by two full time adult workers.
This leaves no room for reduced time workers, such as two adults with jobs that allow them to work within the hours that children are typically in public daycare (warehousing in schools).
It also doesn't help anyone that the building codes are very lousy at requiring sufficient acoustic isolation. I'd sure like to not be woken up early / kept up late by the noises of those in other units.