Wealthy consumers love mortgages because it's a cheap way to leverage a huge asset for market-rate returns.
Normal consumers love mortgages because it allows them to buy a house.
Good luck taking that away.
Additionally, while access to credit raises housing prices, there's no guarantee that removing access to that credit will in any way make housing more affordable for moderate- or low-income people. More people would be forced to rent and it's possible that the wealthy apartment owners will engage in rent seeking and rents will just raise to match or exceed the cost of a mortgage.
That will rob lower income people of their biggest opportunity for wealth creation (housing appreciation) and increase wealth inequality.