Hitler wanted to murder populations based on their ethnicity, and conquer Europe. Bannon wants a smaller federal government limited by a more originalist interpretation of the constitution and less immigration. Those aren't the same things.
Bannon ran an unorthodox campaign premised on populism and nationalism. That's quite different.
The harm in equating Bannon and Trump with Hitler stems from the charge not being true. The harm includes the fact that nearly half the country supports Trump, and you are (unfairly) equating them with Nazis. This is quite polarizing for politics, in addition to being unjustified.