In movie metaphor, Old Man Potter finally bought out Bailey Building & Loan when it went under in 2007, and now Bedford Falls is turning into Pottersville.
As far as I am aware, the FNMA/FDMC system only guarantees mortgage loans for properties with existing [appraisable] structures, so borrow-to-build lenders face significantly higher risks and lower liquidity. It's harder to buy undeveloped land and build a hovel on it from local materials, than it is to buy an overvalued 5-bedroom McMansion, built from twigs, bird spit, and a 10-year horizon for its planned obsolescence.