Yes there other factors but height tends to be a prerequisite before they apply. Odds goes exponentially down based on how far from the mean you are. 5'9" vs. 5'5" is a world of difference.
'Success' is also not Boolean. I know very successful short guys who have had zero success with women.
Talking in blasé terms 'oh it is just one of many factors' is like saying possessing functioning legs is just one of many factors a footballer requires. Or having enough petrol is one of many factors in a car functioning properly. It's true but kind of missing the point.
The really interesting point is why people are so in denial about this fact that is so blatantly obvious.
I suspect it is a mix of those without this issue wanting to think their success was all attributable to their own skills, people not liking the idea that something uncontrollable could screw you so bad, women feeling it is an attack (it isn't at all), those who are shorter but not significantly below the mean having success (though reduced) and thinking it must apply to those shorter than them, etc.
Another interesting thing I have noticed is that short men is one of the few remaining immutable characteristics society allows people to mock openly. "What is she doing with that little squirt" is something I heard from a friend recently without a hint of self-awareness.