My reasoning is that it would not be trivial to guess the phone number from my account/name, and to guess my name from phone number (unless someone hacks into the bank's db, in which I'm in troubles anyways). Furthermore if someone was able to figure out that link, it would not be trivial to do SIM swap on Google Voice, it would not be trivial to attack the Google Voice app. Two or three stars have to line up for someone to sim-swap that GV number.
But some stupid bank go further to ban GV numbers. In which case I just don't bank with them.