I do agree actually. And I guess that I'd say there are "swear" words (crude verbal punctuation) and "curse" words in the sense that you describe.
I do know people who think "g-- damn it" is a far worse thing to say than 'fucking piece of shit' for example.