The use of a password only known to you cannot be physically taken from you as your mind controls that authentication mechanism.
And it's not just people standing behind you, it can also be the security camera in that coffee shop. It can be someone looking through your office window with a telescope from the next building over.
Same goes for your phone. There are ample opportunities to see someone type in their PIN code to unlock their phone, especially if you use it for payments which are generally done in public places. No need for violence, no need to draw attention to yourself, just watch them enter their PIN and then pickpocket the phone afterwards.
It all depends on the exact threat scenario, but biometric authentication can be preferable to passwords when used in public places.
Maybe a bit weird if you start doing that in trains and coffee shops though.
On the other hand, in a few countries police can force you to use biometric authentication, but not to release your passwords. In the end, you really need to think about your threat scenario and act accordingly.
If people home-jack me at night, my 24/7 alarm system/monitoring company calls me in the following 45 seconds at most and asks me for my password. If I say "monkey" it means everything is fine, if I say "beetle" it means I'm under duress. When I give either password, the company answers: "OK, sleep well, all is good". But in the later case they call the police and tell them a home-jacking is ongoing. (obviously my two words aren't "monkey" and "beetle", this is just an example).
(as a bonus my alarm system has an anti-jamming system and communicates using several channels)
Banking apps should be the same: they should have one PIN to do regular business and another one where everything looks legit, but you'd only be making fake wire transfer or only allowed tiny withdrawals, showing a small balance.
Some companies (for example my alarm system) and websites (very few but I've seen some) and some HSM (for example cryptocurrencies hardware wallets can decode using two keys, one of them showing a smaller balance than the real one) have seen the light and have such a feature.
I do believe we're still in the stone age when it comes to security. Most people like to post that disastrous XKCD and think the bad guys have forever won thanks to their $5 wrench. I'd hazard a guess: people thinking with that victim mentality aren't the ones coming up with better security systems.
Who show up three hours later and shoot your dog.