I get where you're coming from -- criticizing people for what they are, rather than for what they do, is exceedingly unfair -- but I can think of at least three important differences, chief among which are:
1. The shortness of men (barring outright dwarfism) has never been the object of widespread theories about their fundamental inferiority, nor have such theories been enshrined into widespread law.
2. Social institutions have never explicitly conspired to marginalize short men.
3. Redlining, lynching, the selling of persons into slavery, ghettos, etc. have no equivalent in the realm of height.
And to be frank, it's rather shocking that you would suggest otherwise. I don't doubt that you've been treated unfairly, and you are perfectly entitled to complain about it, but that doesn't require you to twist reality.