> You can't take the mortgage with you
One wonders perhaps, why is that exactly? Why can't you trade one bit of collateral for another similar bit of collateral?
You owe $300k backed by a $700k house and you want to swap that for a $500k house, why should the note holder be able to call in the loan when you do that?