With physical IDs there's an option to spread them out. I typically keep passport in hotel room, hotel key card in my back pack etc.,
Perhaps Apple Wallet will act as a secondary but a fast path access and I'll continue to carry physical IDs but don't have to pull them out for most of the cases. That is how I used Apple Pay. I carried bank card with me but used Apple Pay 99% of the times.