For retail mortgages there are presumably regulatory considerations as well, eg, the mortgage probably has to continue to be serviced by a regulated entity. It's not uncommon for the original lender to sell the rights to the mortgage but continue to service it.
10 year fixed rate contract, loan of 1 million and the rate falls from 3% to 2% on the second day of the contract -- if you terminate the contract now, you owe the bank 1% of the sum of whatever your loan balance would have been during each of the next 10 years.
To be fair, you get the opposite deal if the interest rate rises.
Better to go to a broker who shops your loan among dozens of lenders to find the best terms. You typically end up with some entity you've never heard of (not a known bank), who will hold the loan for a month or two and then sell it off to someone else.