But still, it does manage to get abused. Unfaithful relatives / spouses / colleagues / etc. can manage to get hold of your password and device, take out loans or buy stuff, and you're 100% in the jam for it. We get cases from time to time where people are basically held accountable for hundreds of thousands in credit/consumer debt, because someone used their signatures to take out those loans. And probably 99 / 100 times, they lose in court, against the banks.
The banks will argue that if they were held responsible for such actions, the modern fast-tracked system would halt to a grind. It'd be like in the old days where you needed to show up in person, with all your financials, and carefully go through everything just to get a small-ish loan.