I think this is a good authentication model, but it costs money. There is the upfront cost of the physical card, and then the higher cost of lost account recovery. I think that's the turn-off to most banks; they will have to staff a call center that can verify your ID, issue a new card, and then deal with your immediate concerns because you can't access the website. Passwords + security questions are often free, "oh you don't know your password? we'll just email you another one if you remember your first elementary school's name".
At the end of the day, they are allowed to cut costs on critical infrastructure (everyone's money!!) because we have a strong victim blaming culture in the US. If your password is guessed, it's because you're bad at picking passwords, not because passwords are an intrinsically flawed technology. If your money is lost, it's because "it's really cool to have transactions that can't be reversed, you should have done your due diligence". It surprises me that everyone is OK with this.
(Incidentally, the only place where I've seen the option to use cryptographic authentication is on Vanguard. They added it right about the time we were testing security keys / U2F at Google, and they were the administrator of our 401k plans. I think Google strong-armed them into implementing it! Would love it if they did this to some banks ;)