The downside is that widespread usage of credit cards increases prices by 2-3% and disproportionally impacts those who can't get a credit card with the same rewards. You are getting a 2-3% discount on the backs of the poor.
Recent profile on one program and the kind of gatekeeping people face: https://www.youtube.com/watch?v=wJDk-czsivk