I would concur in your take on the argument. I think it's important, to promote rational and productive discourse, to push back on those that would have us believe that ideas are in essence, "guilty by association", based on the reputation or history of those who formulate said ideas.

Instead, the ideas, theories, philosophies, etc. should be confronted directly on their own merit or lack thereof. I don't think someone, in good faith, could claim that the ideas of liberty and freedom proposed by many of architects of the constitution of the United States are lessened simply because some of these people were slave owners. That's not to say that we can't look back now and agree that their practice of slavery was morally wrong, but to pillory their positive contributions to the world simply because of this seems quite nefarious.