There is a case for legacy admits - continuity of the culture in a university. The culture of a university is what makes it attractive for the next generation of students.
That said, i support legacies because these private universities at just that--private institutions, and the first amendment gives you the right to associate with whom you wish.
I'm not a legacy admit (my admission was denied, the fools!) but my dad definitely imparted some of MIT's culture on me before I left for college.
Professors of course contribute to the student culture too, but by no means as much as the students themselves.
https://law.justia.com/constitution/us/amendment-01/10-right...
There is generally considered rights held by (among other things) highly selective organizations to not have to follow public accommodation laws (which might harm their selectivity).