Epigenetics allows a much faster rate of change, especially in structures which are not well preserved in the archeological record. For example, cortical folding patterns.
And "how big the brains" is a very good question. See https://blog.frontiersin.org/2021/10/22/when-and-why-did-hum...
> Studying computational models and patterns of worker ant brain size, structure, and energy use in some ant clades, such as the Oecophylla weaver ant, Atta leafcutter ants, or the common garden ant Formica, showed that group-level cognition and division of labor may select for adaptive brain size variation. This means that within a social group where knowledge is shared or individuals are specialists at certain tasks, brains may adapt to become more efficient, such as decreasing in size.
https://www.frontiersin.org/articles/10.3389/fevo.2021.74263...
on the behavioral, there is strong evidence that phobias can be inherited, for example.
> We aren't selecting for intelligence.
I wasn't really focused on the brain size aspect of the discussion.
A bit of googling for backup reveals that things may not be as simple, but it is not clear that current societies are selecting for brain size.