I think this probably is indeed the case. Although I would argue the value of an idea should be entirely detached from who had it, and be a function only of its usefulness, beauty and other significant attributes.
Of course if a Nazi has an idea we might then argue it becomes less beautiful because of that.
Instead, the ideas, theories, philosophies, etc. should be confronted directly on their own merit or lack thereof. I don't think someone, in good faith, could claim that the ideas of liberty and freedom proposed by many of architects of the constitution of the United States are lessened simply because some of these people were slave owners. That's not to say that we can't look back now and agree that their practice of slavery was morally wrong, but to pillory their positive contributions to the world simply because of this seems quite nefarious.
But, basically, the question we should ask:
"is there a direct connection between Heidegger's philosophical prescriptions and his fascism?"
If so, that suggests a pretty serious problem with his prescriptions -- that, in certain milieu -- his ideas comfortably co-exist with or perhaps even feed/justify a fascist mindset.
IMO that line was always kind of obvious, but in any case, Heidegger drew it directly for us in his Black Notebooks.
Especially in Heidegger's case -- because of the very broad nature of the prescriptions that follow from his work -- I think that's quite different from mere "guilt by association".