I think the important thing to realize here is that it's just someone's opinion. It's not a scientific fact that you are the average of the 5 people you spend the most time with.

The theory breaks down when you don't spend time with anyone, because you can't divide by zero!

A live human being is no more influential to me than a good book. If you really want to get a good "5" and you can't find enough talented people, consider replacing your unwanted friends with dead trees.