I think women liking tall men is pretty much universal which would lead to them getting taller and taller though evolution. As they men aren't all extra tall there are presumably counter factors like tall people needing more food in famines or having back problems I'd imagine. I heard someone say the average Frenchman used to be taller but Napoleon put a lot of the tall ones in his army and then lost them in battle.

It's important to note that women's preferences is not the only thing affecting evolution - shorter men can be selected due to higher survival rates (for example in jungles, as another commenter pointed out).

This is just speculation but I think most likely in a Jungle environment where being light on your feet helps during a hunt.