"By giving such speakers a forum, they're endorsing that such views are worthy of being heard or debated"
Exactly. Free and open debate of controversial ideas, however heinous, say in a classroom setting is very different from actively providing hate-mongers a prominent forum from which to spread their hate. Former is academic freedom, the latter is a political decision.