I used to think like that too but I think it’s flawed. By that logic you should withdraw all your money and pile it up next to your passport and unencrypted drives because they could pilfer your drives to get bank passwords anyway.
Just because something doesn’t offer complete security, doesn’t mean it’s a bad idea.