It's actually well accepted in biological research, plenty of evidence for it, surveys of women, OK cupid had a blog about it (before they wiped all that).
Also all of literature would suggest otherwise. Look at any documentary talking about a man who is considered attractive and see what word they use first. "Tall, dark and handsome" - notice the order. The list goes on and once you start looking for it you'll be surprised. Look at romance - novels - see how many of the men are short.
You could also speak to some women and ask them about it. If you ask enough you'll get at least some honest responses (not disparaging women, equally men may often be deceptive on this kind of question).
I'm also not sure I understand your argument - both men + women contribute to height, male height is only part of the equation (my dad is normal height my mum is very short and I am short for example). Also nutrition has been HUGE in stunting growth, societies with better nutrition have a very big increase in height (which also suggests sexual selection plays a role I'd say).
Height is also in a pretty narrow range anyway, +-2SD.
Finally I'd say that if you were significantly shorter than average and male + had tried your hand with dating you'd have a very raw and immediate series of data points that would be pretty compelling.
This isn't really a controversial point, in the same way men have a preference for a certain hip/waist ratio in women ("why don't all women then have that hip/waist ratio" you might ask following your existing argument...!)