That's true except for cash out refinancing. Lots of middle class folks take out home equity loans to pay down credit card debt. This was a big component of the 2008 financial crisis. That and banks leveraging 40-1 and betting big that housing would never decrease in value.