Do you not also keep your ID on you at all times?
That's why I don't mind carrying around plastic cards, they just go in my wallet with my ID and a small amount of cash anyway.
Even if I was willing to use GPay, I don't like the possibility of my phone dying locking me out of my apartment and preventing me from paying for things.