This is true! This is why I never ask for feedback. I am completely silent whenever observing a playtest. Every change that I make is purely based on the problems I see my students and co-workers experiencing.
It is hard to do this when asking the question, "Was that fun?" But I try to answer this question by seeing player reactions. For example, many students think it's funny when all the bananas fall out of the sky on level 2. The original design goal for this level was trying to dig physics objects out of a pile of other physics objects. But the absurdity of that visual usually gets a couple laughs.