It "the" factor as in: the most important out of other factors. One you will often get automatically disqualified for. One for which you will be automatically taken less seriously in dating, in other social situations in workplace in politics. One you can do absolutely nothing about. Other things may make significant difference but most things are possible to work on. With height you are out of luck and have to listen to condescending "short guys should put more effort to deserve the same" talk your whole life or my favorite: "maybe you should lower your ambitions" (often disguised and said in more pc way) - why ? Because you were born that way.

Now, being the most important still doesn't mean it's 80% or even 50% if you try to measure those things as there many other factors (and sucking on many fronts will reduce the biggest of inherent advantages) but its huge.

You said it's not the most important factor but then when challenged you offered logical fallacy as an argument which makes probability that you never really thought about it or experienced it quite high. It's difficult to assess from within the bubble so what about reading posts of people who experienced it here, at other places on the Internet or looking at the studies which were done ?

/rant