Why would it be harder for a student from an elite boarding school to get into an Ivy than someone from a rural school? I would say the opposite, in no small part based on the evidence that at least supports a strong correlation.
If this is news to you, I suspect you haven't thought much about these issues.
I'm admittedly unfamiliar with Ivy League culture. What does fencing have to do with your application?
Under the rule of the league they can't give out athletic scholarships, but they can, and do, recruit athletes and subject to a less rigorous admissions process.