Not with Google Wallet.
It requires that the card issuer support Fidesmo though. Many here do but I'm not sure what it's like elsewhere.
To be able to do it, you have to authenticate with your card issuer in a mobile app, similar to how you might when setting up Android Pay or Apple Pay. The mobile app then uses your phone as a bridge between the issuer and the NFC chip in the accessory so the relevant data can be written in a secure way.
But not Google Wallet, and with GrapheneOS as the connected device?
At least one of my cards required Google Play Services to have the location permission when initially adding the card though.