I think regular human size can vary a lot by level of resource consumption - the principle being the larger the mother, the larger fetus and so the larger the resulting adult.

Obviously, if a height increase/decrease trend continued for a while, one assumes genetic variations which aided this would be selected for but it seems that the change itself doesn't necessarily need genetic as such.

See height increases in Japan: http://www.nytimes.com/2001/02/01/world/tokyo-journal-the-ja...