I don't know if it's my bias speaking, but I felt that in the debate, Buckley comes across as a bully, with very little argumentative meat, who doesn't really respond to Baldwin's points. For what its worth, I didn't really like Baldwin's delivery either, but at least I felt there were substantive points there that were worth addressing.
In a way, this is probably the best one can hope for when you stage this kind of event - Buckley is, at least, well educated, but I can't help but think the debate might have been far more interesting if it was Baldwin vs Malcolm X - because ultimately the problem with the Buckleys of the world is that they defend institutions that were not built on ideals, but rather economics, and as such the defenses themselves are generally sophistic and without anything really interesting in them.
If I was a student, I would probably agitate to exclude a Buckley, and instead reach for a Malcolm, and I expect somebody would accuse me of being anti-free-speech, since the debate that would result would no longer span a societal divide, and both parties in the debate would be on 'one side' of the broader social issue. However, from an intellectual standpoint, it just sometimes is the case that interesting positions aren't evenly distributed across the terrain of popular feeling and political rhetoric, and a lot of people are simply defending things that are convenient or traditional but have no moral basis. So inviting those people is a waste of time - they're just (at best) going to come up with clever excuses for whatever already exists, or whatever they would want regardless.