After a few weeks off of dairy our memories of those old textures and mouth feel start to fade and are replaced. Humans don't like changes in patterns, but at the same time are relatively quick to adapt to new ones.
There's real addiction at play with dairy as well [1]. Those young calfs get the dopamine hit to attract them to their mom's milk. For humans, it fuels our addition to dairy, whose importance as a food group was manufactured because it created another revenue stream after post-war food industrialization [2].
[1] https://www.forbes.com/sites/michaelpellmanrowland/2017/06/2...
[2] https://www.vox.com/2015/4/19/8447883/milk-health-benefit