There are virtually no fixed costs to actually performing the transactions. The fees are levied by the not-so-competitive Visa/Mastercard/AMEX trio that would love to keep their fees artificially high.
They have to pay for infrastructure, they have to pay for customer support, they have to pay for fraudulent transactions and on and on. I think most credit card companies have something like 10%-15% ROI
This is why large e-commerce companies have giant fraud departments trying to prevent fraudulent orders and charges on their site; because when they occur, they generally just eat the loss.
From wikipedia (http://en.wikipedia.org/wiki/Credit_card_fraud#Merchants): "The liability for the fraud is determined by the details of the transaction. If the merchant retrieved all the necessary pieces of information and followed all of the rules and regulations the financial institution would bear the liability for the fraud. If the merchant did not get all of the necessary information they would be required to return the funds to the financial institution. This is all determined through the credit card processory."
From what I have seen, it is very common for the merchant to be liable for losses on a card not present transaction (e.g. online retail) but the credit card company stomaches the losses for a card present transaction.
I suspect that online merchants could reduce their liability by implementing things like Verified by Visa but choose not to because it causes them to lose too many legitimate sales.
Blame the banks issuing credit cards for the high rates, not Visa and MC (both AmEx and Discover are, effectively, their own issuing banks - so go ahead and give them their fair share of the blame)
I work in the industry; it's my job to know how this works. Here is an example from Wikipedia (bear in mind the amounts are fictitious but relatively accurate): http://en.wikipedia.org/wiki/File:Gao-report-on-interchange....
To address your point, I understand interchange as being set and collected by the card network, and is non-negotiable. Anything off the top is left for the banks to keep for themselves, e.g. the plus in "interchange plus" pricing). Another example assume a hypothetical interchange of 1.5% and a merchant that has a merchant account that costs 3.5% + 30¢ (gateway) per transaction. In this case the card network takes 1.5%, the gateway 30¢, and the banks 2%. Is this not how it works? It would be nice if you could cite something (not trying to be an ass in case it reads that way :).