It isn't just the card brand taking a slice.

At best it's a 3 party system between the Gateway (Auth), Acquirer (person actually paying the merchant), and the card brand. I don't think it's uncommon for their to be middle-men between the acquirer and merchant.