The particularly uncharitable reading of that quote that jumped to my mind when I read it is "but why can't I, a white guy, say the n-word?" But this probably isn't what Graham actually had in mind.

I think Graham is trying to operate in some idealized plane of pure logical statements where you can speak truthful axioms and reason from them. He misses that in the real world statements have a context, are part of an ongoing cultural conversation, and imply consequences.

There are things you can say which are "true" but which miss the point or suggest that you're pushing for a certain policy outcome. There's a lot of situations where you can say "the statistics say X and so we should do Y" and the (fairly valid) rebuttal is, essentially, "why did you accept the societal structure that produced those statistics?"