Saying only 'impairment' implies that all attributes associated with autism are negative compared to some mythical baseline, and creates a circular definition which stigmatises autistic people.
The ones that are diagnostic (except those that are about ruling out other diagnoses) are. If we understood the mechanism enough to identify anything other than observed dysfunction, that might be different.
And, yes, like virtually every mental disease, the negative attributes are defined in reference to the concrete social context in which the individual exists, and could equally be viewed as problems with society. That's a problem for a different domain than the medical though.