I'm a small business merchant, and they're passed on to me. I pay interchange plus 0.04%. If interchange on a specific transaction is 0.05% then I pay 0.09%. If it's 2.9% then I pay 2.94%.
Merchant service providers have been pushing this "interchange-plus" pricing model to win over businesses from other providers with the old "tiered rates" pricing for many years now. Even "tiered rates" did pass on some of the price differences, just not as directly.
PayPal/Stripe's model of a single flat fee for all card types is pretty much exclusively an internet (card-not-present) thing, except for the rare coffee shop with one of those Square smartphone dongles. None of the B&M stores you shop in are likely paying flat rates. Different cards cost them different amounts of money.