Paying with credit card is in itself rare in most of the world outside the US, where debit cards and cash are way more common. My guess is, 95% of all locals and non-US visitors automatically pay with cash and don't even notice the Visa-only signs.
Not having the local currency on you and waving your credit card in situations where it's considered inconvenient and inappropriate is a very typically American tourist thing to do.
And I really can't think of anything within the Olympic venue that would be considered appropriate to pay for with credit card.