This is one of the big issues with MFA: you should ideally never create a single point of failure in the second factor, so you want two hardware tokens, a hardware token and a fallback TOTP, etc., but also ideally the ability to have fallbacks that do not rely on email or SMS.
Most websites including financial ones fail on the second point. And it worries me because SIM swapping is real and does happen to people you know.