I understand that this idea of "adding anything destroys the flavor" is popular and widespread, but it's a really odd thing to think. That's not how cooking or taste buds work, so why would it apply especially to coffee?
The idea of buying expensive coffee is to taste the coffee, not the cheap sugar, milk or chocolate.