I think more people that go to Ivies are hard workers then have so much natural ability that it doesn't require much work. Nowadays you have to be doing multiple extra curricular activities and qualifying for summer programs, etc.
Even the not rich kids often have had very supportive families. The vast majority of students don’t take college classes in high school / decide to compete in olympiads by themselves. At the least it takes a supportive teacher or more likely a supportive parent to suggest the idea—most kids in most neighborhoods aren’t even aware of the existence of such programs.
rich kids might go to less-prestigious private schools, but they're not getting into the Ivy League without working hard at something.
Or not. I've been told by someone teaching introductory courses at an ivy, that a major challenge they face is that their students have had such good teachers, presenting ("spoon feeding") information so clearly, that their students now have neither the skills, nor the inclination, to wrestle with a body of knowledge and extract understanding.