Also in classical times the 'seat of power' for men was not the penis but the testicles.

Today we equate masculinity and virility with a large penis, but to the Greeks and Romans a large penis had no such connotation.

When you think about it, it makes sense from a biochemical perspective. Without the male testes a man can not produce testosterone which is the key hormone that regulates growth, muscle mass, aggressiveness, competitiveness, vitality, etc.

So it is understandable to think that a larger pair of testes would make you 'more' of a man and a smaller set, less.

Not sure when exactly this change occurred in Western culture but it would be the more interesting question to try to answer.