I've always wondered if these cash transactions even make it onto the books, I suspect that many of them are not only merchant fee free, but also untaxed.
For a meal, that's fine. For a $2000 computer from a shady seller on the streets of Hong Kong, I would gladly pay $40 as an insurance against fraud. Peace of mind is nice and good CC companies (American Express) got your back.
Your reputation is worth more than anything else in a small community. If you start ripping people off its known very very quickly and scammers end up moving away, or just get well known enough that nobody sane does business with them. Eventually bad enough apples get encouraged by the local brutha's to stop being a problem or have their limbs broken.