It harms people when there would be a different group of "most talented and skilled people" if they were given the opportunity.
Bill Gates and Mark Zuckerberg had great ideas, but they were also rich before they made a dime of their own money, and they never would have gotten anywhere without that money and their connections.