In "Sapiens: A Brief History of Humankind," Harari claims that human brain size has decreased since the agricultural revolution (which fits the whole downfall story of course).

A bit of googling for backup reveals that things may not be as simple, but it is not clear that current societies are selecting for brain size.