Yes, making society worse off, knowingly: that means harming others: yes, be ashamed.

If you're no longer convinced of that, it's because you are looking for justifications for defect in the prisoner's dilemma, rather than cooperate.