Historically, your phone was a 2nd factor, but not anymore. Why do app developers ignore this basic thing?
1: Person must have physical access and control of your phone.
2: Person must have Phone pin-code or password, in order to unlock phone and access either SMS, banking app, or password app. Or somehow brute force password or decrypt phone contents.
3: Person must have either password for banking app or password to Password App to gain banking password.
If you're not using passwords (and different ones at that) for your phone, password manager, and banking apps, then that's the whole in your security, not from using SMS as a second factor.
There are issues with using SMS in 2FA, but it's not the issues that you've brought up.