The problem is that natural investment opportunities with a given risk profile are a limited resource.
A healthy economy would make investors compete over those investment opportunities. If a lot of investors want to put their money into residential single-family mortgage loans, and only N families seek to borrow with that type of loan per year, the investors would have to offer lower interest rates for every level of default risk. This limits the amount of money the investors can make by lending, so some go off to look at different types of investment. The rate of return naturally sorts itself out such that it increases with increasing risk.
When rates of return are low, entrepreneurs generate more investment opportunities at all levels of risk. But housing starts and transfers are more closely associated with employment opportunities, which tend to trail new business starts. People don't buy houses unless they have jobs that pay enough to buy houses, and they don't have better jobs unless businesses are expanding and labor is getting more scarce.
What seems to be happening is that middlemen are assembling artificial investment opportunities with a falsified risk profile, in order to capture investment dollars that would otherwise leave to find other opportunities with a better return/risk ratio. Some of those investments would have eventually sloshed back into mortgages, filtered through new employees or promoted employees, to replenish the pool of available investment opportunities. So the risk fraud is self reinforcing; the more it happens, the more it has to happen in order to maintain the status quo.
If the financial institutions are completely serving the mortgage market, that excess money has to go into other investments, or it loses its value. They won't be mortgage investments, but there are only so many of those around to pick up. The "nightmare scenario" is "OMG! We need to find another parking spot for our colossal pile of money! And the available ones are SOOOO FAR--like 50m away!"