I'm not convinced on this - with a large enough student base (e.g. most state schools), I think it's possible to find peers that would have excelled even at top 5 universities. Similarly, at top tier colleges, you can find people who don't really perform to the expected level and coast along.
You stand out by being a top performer, and will attract other top performers as long as you search them out. Also, in those scenarios, professors will give you more attention, because you are a better student and more enthusiastic than most of your peers.
Perhaps if you are a true prodigy, you might need a top-tier program to reach your full potential... at that point your peers are the professors and high-performing graduate students, but for most students, I think there are pros and cons for being in a top 5 vs a top 50 program.