If someone steals your phone (you will lose at least the value you paid for the phone) and once they force you to unlock the phone and to hand over the passwords, you will lose:
- as much cash as they can take out before you can access a device that will allow you to freeze the card
you will also be a more attractive target for hacking (especially if you are using Android and your phone is like the vast majority of targets that receives security updates very late or it simply does not receive them)
Seen it happen more than once. Yes, you are smart but so are thiefs
Also how do you do anything if your phone is gone - travel, call, let the bank know, buy a new phone, let your partner know you're in trouble, anything?
Also how much is you phone worth? A quick look suggests ~£1000 for an iphone 11, although I guess that's needlessly high, but you're carrying a large cash equivalent, highly attractive? I've never had £500 or the equivalent in my pocket, ever (max ~£350 for 1/2 hour IIRC).
And cash doesn't break if you sit on it. I'm not entirely convinced of your case quite yet.