>improved memory is absolutely something evolution would select for in primates, especially humans.

What if it makes people more scared, because they can remember past experiences better. This could lead to less procreation and thus inverse selection pressure.

Only because a trait is superficially good it doesn't mean nature would select for it.