I think the concern is more that it is a lot easier to accidentally lose a wallet or purse (or have it stolen for unrelated reasons), rather than a targeted attack against your person for the master password.
And, as I mentioned earlier, we're really quite good at keeping track of wallets and purses in the first case. Losing them is typically (at least in my own experience) a once-in-a-decade event.