You are better off opting the expensive coffee with the alternative options for lactose intolerant people (soy milk etc.) when compared to buying milk from the road-side stalls (right?)
I have been having coffee almost daily for the past three years and I see a significant improvement in my bones and joints. I can feel it and pointed out by the doctor. Note, I take the coffee with whole milk and not the half-and-half or other kind.
So I would guess it does affect in the long run? Personally it did for me. Not sure at a Macro level.