"Enemy of the people" is also not inherently illegitimate - I mean, the people can surely have enemies, right?

But words and phrases like that have a history of actual use and the resulting connotations.