Poisons in very low quantity are used to treat diseases. Could it be that Ayurveda uses milk to treat the imbalance of the "doshas"?
As a food, Ghee was used so very sparingly that it was almost not used at all. We cannot compare the quantities we consume today with those before the Dairy Revolution.