Specifically, they pass it on to all their customers, not to only those customers using cards (doing so is against the merchant-processor agreements). i.e. those using cards are in effect increasing the price for those not using cards.
It's a massive problem for businesses with lots of small transactions, if you're selling a £2 coffee and the CC processor charges 30p + 5% per transaction. Previously nearly no-one would pay the extra, now you have to cover it; so you make the coffee £2.40 ... but the large retailer with a blanket 3% transaction cost (or whatever) can undercut you by quite a lot.
https://www.theguardian.com/money/2018/jun/17/credit-debit-c...