By constraining the supply of loans, first and foremost you’d be cutting out those who need it most. Unfortunately, that’s how risk markets work - the riskiest (also usually the poorest) get cut first.
The interest rate argument isn’t actually very grounded in fact - in order to save from refinancing, the rates have to have declined from where they were before. If rates stay flat, there is no benefits in refi - whether they stay flat at 5%, 3% or 10%, as long as the rates stay flat. However, borrowing at a lower rate is better because you end up putting more into the equity. If you borrow at zero percent, everything you pay in goes to reduce the principle and so long as the property doesn’t decline, you are building wealth.
This is why arguing to reduce the loan supply to me is very counterproductive. What we need is better education and fewer predatory lending practices. Things like payday loans, etc, are currently predatory. Those could be reformed to serve the purpose they were originally intended for (emergency money), but until that happens, things like that create the actual problem, not the loan supply itself.