There was a study last year that pointed to Dutch women preferring tall men as the most probable cause of the rapid increase in height. [1]
[1] http://www.sciencemag.org/news/2015/04/did-natural-selection...
[1] http://www.sciencemag.org/news/2015/04/did-natural-selection...
Women still prefer many of these physical traits, but probably not as much as wealth, which is linked to intelligence, which comes by default with age.
So you have to factor in what traits are conducive to wealth accumulation, and the easiest and biggest one is probably longevity. If you can be an older, wealthier male with his shit together, that's probably one of the bigger factors over height.
How many short men are getting excluded from gene pool as a result of this?
My take: almost none. They might get "less desirable" women, but otherwise all men and all women are going to pair up.