There's a large amount of literature on the effects of diet on meat flavor. I'm much more familiar with that side of things than milk flavor, but anecdotally I would expect similar effects.
Re: not consuming dairy, you be you. Cheese is rad. Good beef/pork/lamb/rabbit/etc. is rad. I think it's entirely possible to produce them in a sustainable and humane fashion, since I've personally been involved in producing beef that way (registered black angus FTW). My moo-cows had happy lives until their end. I think that's an attainable goal and compatible with also enjoying a wonderful tri-tip cooked with oak wood.