I'm not sure 'a doctor' is necessarily the right person to consult, but lack of it can induce lactose intolerance, and it's a good source of calcium which children have greater need of than adults (so a dairy-free diet that's been great for you for 20y isn't necessarily great for your child), and perhaps other vitamins, IANA expert.
And since GP said infants, if we're including (human) breast milk as 'dairy' then it probably is reasonable to ask doctor about it - IANA parent either but I think the standard powder or whatever substitutes are still (non-human) dairy products?