So was it more damaging that a small percentage of cars were cheating on their emissions? Yes. But how much more damaging? Not much, especially since diesel cars are a very small fraction of VW's sales in the US.
Overall it is better that this group of cars were cheating as opposed to say Toyota's gasoline powered cars. The overall effect was so small it was not detected by the EPA's and CARB monitoring efforts on air quality which they model against the assumption that people are complying with the regulations.