That it can be a fallacy does not mean it is always a fallacy.

Sometimes, guilt by association is warranted.