From my own anecdotal experience, lower inflammation has a wide variety of cognitive benefits, regardless of the means, while things that increase my inflammatory response (for example, allergies), make it much harder to think clearly.
Side notes: I don't drink milk so can't speak to the lipids that might further influence intake. But even the heat of coffee will have an effect: if it's scalding hot, some of the caffeine will be absorbed in the mouth, and if it's very cold, it'll wind its way further through your system before being absorbed.
I'm just a coffee drinker, so take all of this information with a grain of salt, unless your sodium is too high? Caveat emptor