That kinda ruins the main utility of the system for most people. People aren't switching service providers often or SIM sizes at all these days (the latter especially with larger SIMs just being the same nanoSIM but with an adapter of sorts around it).
For the average person, the best utility of it is being able to retain your number on losing your device. Usually you aren't expecting to lose your device, so imagine how much more complicated everything gets when not only do you have to worry about getting a new one unexpectedly, but then also have to get a new number, inform everyone you know about that, and then update all your accounts.
All of that just to protect people from being targeted by fraud that is not only very unlikely to happen to them unless they're well known, but is also better resolved by making authentication systems smarter.