If you bought a house for $150,000 at 15% in early 80's, you likely refinanced as the interest rate continuously dropped from there. In 90's interest rate was below 10%, and around 7% in early 00's. Nobody would keep the 15% over 30 years.
Your payments would total $375k over that period. $350k at 4% would total the same amount but with $200k in principal paid down.
I'd much rather be in a situation with $200k in equity versus $20k in equity. And remember, both people have the same monthly payment amount!