A lot of banks in Germany still offer photoTAN generators. Effectively, a physical device that generates 2FA codes for your login. You can then use the website as usual and use the codes from this instead of phone confirmation. This is one example from ING.[0]
That way you can effectively use most feature phones as your daily drivers. HMD (Nokia) still manufacturers some of them that even come with GPS, etc. There are some feature phones that even run Android but I don't know what app support for things like Spotify is like.
Alternatively, keep a cheap smartphone around with nothing but the banking app on it.
I wouldn't necessarily mind a dedicated piece of hardware, but if it's dedicated hardware, I feel like the bank ought to provide it, not make me pay for it. And I'd much rather it be a generic piece of hardware that can also be used as a second authentication factor for other things, like a Yubikey. I've looked around but have not found any banks that offer this. The only one I had heard of no longer offers it.