As humans we have always tried to obtain foods that are safe and nutritious. Even though milk has potential allergens it contains no poisons, toxins or anti-nutrients unlike many plant foods. It is incredibly nutritious and contains sufficient amounts of most minerals and vitamins needed to sustain life (babies use it exclusively without developing any deficiencies after all).

Apart from this it has other qualities that make it useful in a post-agriculture world. Diets rich in grains and animal meat are too rich in iron and phosphorous and deficient in calcium. Milk is low in iron, and even binds to iron reducing its absorption. It also has higher calcium compared to phosphorous. It has been shown that the ratio of calcium to phosphorous matters more than absolute consumption of each nutrient, and should be slightly higher than one if we want to avoid leeching calcium from bones. Diets rich in leafy greens and fruits have a favorable ratio, while meats, legumes and cereals do not and this probably contributed to the skeletal deformities we see post agriculture. Dairy helps shift the ratio closer to optimal in the presence of phosphorous rich foods in the diet.

With allergies and intolerances being the only reasons not to drink it, and those being more common in poor health. The question should be, why not drink milk?