Yeah, my impression is that it's predominantly not used to accurately describe the political leanings of a person or group of people, and it's just a quick modifier that can be used by people to paint the target as an extremist who holds anti-social views.

If I were to wager a guess, I'd guess that the targets of the term 'alt-right' would rarely accept that description of their politics. Whereas 'right' or 'conservative' or 'social conservative' or similar terms would have a high acceptance rate for those that are being called it. 'Alt' has become a pejorative in everyday use (at least in the liberal circles that I run in).