1) To know about Harvard and elite universities in general. Believe it or not, research shows that most qualified poor kids don't apply because they don't have any knowledge about the value of an elite education or that places like Harvard exist. Anecodotally, I know a couple of brilliant ones from poor backgrounds who went to their state universities simply because they had no idea there were other options - in fact, it was an imaginative stretch to go to college at all.
2) Family and community support for a college education, much less an expensive elite one far away. Sure, some super-teen can overcome this, but it's a major reason kids don't apply or drop out at high rates. My two friends were constantly asked, why would you want to go to college? Kids in these situations are in alien environments at college (that it seems perfectly normal to white middle-class kids should tell you exactly what environment it is) where they have few people who can relate to them (Edit: and an institution not built for their needs), then they go home and are asked why they are wasting their time and family's money. And as they absorb university life, they also become alienated from people at home who don't have those life experiences or opportunities.
3) Money: There is a very strong relationship between family money and college education.
4) An excellent high school education: Something else that correlates strongly with the family they come from.
5) Connections: It's not what you know, but who you know. Do you think all those kids in Harvard are there on merit? What about the legacies? The big donors' kids? The kids with recommendations from alums? Those aren't avenues available to most kids.
And all these things have a strong racial component. If Harvard can eliminate these factors then sure, why not eliminate affirmative action too?