I agree, but that's kind of what we do to resolve moral cognitive dissonance all the time, with varying results. Personally it's something I really struggle with, and the questions usually boil down to whether I would be fine living my life wrongly, or if I'll give up enjoyment of my already sad life just to feel good about myself for being moral.

I just can't picture it. 9 hours a day in the office, then I come back home and I just want to hedonistically enjoy all that I can until I sleep. Of course I'm going to want to eat a big steak, watch porn, and listen to pirated music, morality be damned, and I know it makes me a bad person.