Fraud isn't one big problem, it's the biggest problem. Fraud is rampant. And no ability to reverse transactions means that fraud will keep going until no one trusts the technology at all and we are back to square one.
Without trust, it just won't work and that's where the credit card companies did it right by guaranteeing against fraud. Of course, they did this off the backs of the merchants instead of putting their own money at risk but that's a different discussion.