That's exactly what ethereum is - you're able to move funds without permission, trustlessly, instantly.
Banks have fraud experts on staff, they have people that can monitor activity and stop such transactions. Both sides have accountability so that the thief can be tracked down. Your worst outcome is getting your card skimmed and not noticing it in time to report the fraud unless you deliberately send your bank info to someone else. But even then, your bank can probably still help you.