Maybe it was more desperation from waiting for cheese or something? I'm guessing if they kept milk cattle and didn't eat it they had a reason?
Cheese seems like something that cows would be used for. It's easy to make, it comes from curdled cream, it stores for long periods, and it's calorie dense.
Maybe milk drinking came at a time of extreme hunger where people just drank it directly instead of waiting?