Same. For me, dairy increase respiratory mucus production which makes me feel a little congested and increases the frequency amd severity of upper respiratory tract infections.
It also hurts my digestive system.
Both are subtle initially and ramp up with continued dairy consumption, so I can tolerate small infrequent amounts.
I suspect many more people have similar issues but don't realise.