The downside of going cashless aren't hypothetical or as far-fetched as people think. The whole society can be put on hold overnight. We can see this happening in front of us.
Using whatever payment method for convenience in itself is fine. But, eliminating cash as medium, without providing a better alternative, one that can still work in disasters where power or network is down, is wild.