Rich kids from well-connected families take it for granted. Why do you think parents in New York pay $40,000 per year from pre-school to high school? Because you're an auto-admit at many of the Ivies (Harvard, Yale, and Princeton are still a reach unless you're top-25%, but there's always Stanford) if you attend one of the top 5 prep high schools.
Also, Ivy League is one of the less important elements of pedigree. Pedigreed (read: well-connected, not necessarily well-educated and certainly not cultured) people who don't go to elite schools still have their advantages. Also, the middle-class smart kids and the pedigreed legacy kids tend not to mix at elite schools anyway. A prestigious university provides a bump, but not as much as one thinks. It's not like going to Harvard makes you set for life. I know Harvard grads making $50k as social media whatevers at dipshit ed-tech startups. Middle-class people tend to overestimate the connections-forming capacity of college. It certainly happens, but the tendency for 18- to 21-year-olds (well, everyone, but especially adolescents) is for like to group with like and that means that the middle-class smart kids and the dumbasses-with-family-connections don't mix.