Perhaps less "weird cultural phenomenon" and more "it's about the same size as a credit card", plus "it's probably the only broadly accepted government-issued ID you've got, so there's a variety of things things you might need it for, some of which you might not expect when you leave the house", resulting in "just leave it in your wallet or your handbag or wherever you carry such things".
If you're already generally leaving it with all the other miscellany you carry by default, it'd be more effort and more risk to take it out whenever you don't think you'd need it than to just leave it in your wallet/bag. If you're going to buy something, generally you'd have your wallet.
How do you handle such things, then? Honestly curious.