It’s more than that - banks used to have prepayment penalties. People with high credit scores are (probably?) also more likely to move, which means the bank may only get a couple years of interest until you pay it back (when you sell to buy a new house with a new mortgage), vs. people who stay in their house for many years. These things mean that someone with a lower score may be a better investment for the bank.

If it sounds ridiculous… imagine that every month, all the mutual funds you are invested in shut down and gave you your money back (with interest), and you had to choose from a whole new set. Some of you probably do this anyways, but most people prefer to make the choice once and then just let it grow.