Are you also against the terms 'hypocrite' and 'cheap talk'?
By all means, it is far better to use common parlance.
The word is an apt description of something very real and very common [0]: people hiding their greatest moral failings behind ultimately empty virtuous performances.
It's unsurprising that the only people calling it 'loaded' and despairing over the existence of the term, identify with the group it's most often used to describe. And it's unsurprising that they would misunderstand it to be a term that you use to describe virtuous acts instead of sanctimonious frauds.
Desperately wishing that you were able to police the language of your political rivals isn't a good argument for getting rid of a phrase.
Also, if every time somebody uses the term, and somebody jumps in to say "I think you mean 'they're being a sanctimonious hypocrite'" then really it will just be emphasizing the meaning. I don't see what you expect to gain from this...
[0] http://www.thewrap.com/joss-whedon-feminist-hypocrite-infide...
It is interesting that one of the earlier pop uses of "virtue signaling" was on LessWrong, and hence fairly apolitical [A]. But it seems to have been mostly used by the right. If one was to see a lot of leftist groups calling freedom of speech protesters or statue protectors "virtue signalers", then the phrase would indeed retain its original apolitical, universal nature. The article also gives some interesting analysis about how the original term has semantically drifted over the years to become watered down in meaning- at least according to Sam Bowman.
I do agree with Sam Bowman that the term is often misused (and that it doesn't make as much sense as it should since the word 'signalling' is meant to refer to costly signals and not 'cheap talk'), however my belief is that people are critiquing because the term is hitting 'too close to home' rather than because the term warrants serious criticism.
> Here's an analogy, a term I abhor for the
> same reason: "mansplaining".
>
> It's a similarly cheap and unfair argument, because
> it's completely generic and can be used in any
> discussion. It's also impossible to argue
> against, because it uses any answer as proof.
I don't really use the term 'virtue signalling' anymore, but I do get annoyed when somebody uses it and another person jumps in to tell them that they're not allowed to say it because it is used by the wrong sort of person or because it has no meaning. It does have a meaning and it is sometimes appropriate: male feminists who turn out to be using this identity to cloak their misogyny, politicians preaching about peace but consistently voting to bomb countries, Ivanka Trump harnessing a feminist image to hawk sweatshop clothing, etc.
Actually, I feel the same way about 'mansplaining'. There are cases where men step-in to re-explain what a woman just said to her or to others in a louder, more-confident tone and in these cases I think it's a great word.
Edit: You deleted your comment but I had already written this so I've posted it anyway.
But it's quite likely that usage of the latter term has always been a sarcastic jibe about the fraudulent/low-investment quality of the 'virtuous' acts that are being performed.