Think about your usage of the word word people - as opposed to the singular. In a purely random natural selection driven system, propagation relies on the success of a single mutated individual until we get to
very large populations where randomness could more regularly yield similar mutations among different people. Because of this, in the purely random natural selection system the magnitude of an effect is extremely important. Let's imagine we can somehow quantify exactly how likely a specimen is to survive in a certain area per year, we'll call that value x. Imagine x is some percent per year. How much would a random specimen suddenly developing lactose tolerance increase that percent?
In the case of something like Darwin's birds, the percent improvement from beak changes would be very substantial. That mutation would open up entirely new, and vast, resources of food available at all times and completely independent of everything else. In the case of milk, you need a lactating animal to start with. That already implies you have multiple sources of food in extremely healthy shape. That little catch there already ensures that this mutation would not provide a substantial benefit in a one-off scenario.
By contrast, imagine our mutations are not entirely random. And that our bodies, somehow, trend towards more desirable mutations over time -- meaning we don't see just a one-off instance of favorable mutations but them tending to occur over and over again. In this case your language of "people" would be entirely appropriate. Over time, even a small survival edge means a lot and, given enough samples, we'd certainly expect this trait (and any trait for that matter) to start to 'stick' in any area where its net effect on survival was basically anything above 0.