I'm not a historian, but I believe Hitler expressed his anti-Semitic ideas quite clearly in his book prior to taking power. Hitler was involved in violent attempts at revolution, and murdering political opponents and rivals prior to taking power.

Bannon ran an unorthodox campaign premised on populism and nationalism. That's quite different.

The harm in equating Bannon and Trump with Hitler stems from the charge not being true. The harm includes the fact that nearly half the country supports Trump, and you are (unfairly) equating them with Nazis. This is quite polarizing for politics, in addition to being unjustified.