Contrary to what is expressed above, Hitler did much more than give speeches. He also gave orders which is a different thing entirely.
Saying "Gee I wish so-and-so were dead" is not a crime in any reasonable democracy. Telling your subordinate "I order you to go and kill so-and-so" is. Do you see the difference? The person following and the person giving the order are both responsible.
Are you arguing that the people who physically pulled the triggers or shoved prisoners into gas chamber do not bear any responsibility? At any point, all that was necessary to stop Hitler was for people to not listen to him. What would help that is providing an environment of healthy dissent, not stifling it.
I also mentioned above specifically that we should hold people accountable for the actual consequences of their speech. How you get from that to "Hitler did nothing wrong" is completely beyond me. Actual consequences mean real things that occur in reality. Not "I think something might possibly happen because of this" or "You dared to contradict an authority so you must be silenced".