Hacker News Reader

Comment by arnaudsm | Hacker News Reader

Did it fix the model censorship about Uyghurs and the Tiananmen massacre ? Do we have benchmarks to measure political censorship?

View on HN

Any benchmark of political censorship would, invariably, just measure (assuming the benchmark itself was constructed perfectly, though realistically it would only be an approximation) against the benchmark creators preferred bias.

What do you mean invariably? There are some topics that the models refuse to discuss or provide very vague answers. Some interpretation will be subjective, for sure. But you can always check if the relevant facts are presented. I agree it gets muddier afterwards, however DeepSeek doesn't meet event this baseline.

I don't really think so. If a model refuses to tell you anything about a historical event as we've seen in some examples, there is very little bias involved in how to interpret the result.

Even if your entire measure of bias is based on refusals (which is going to be bad measure for other reasons, but certainly easy to construct), there is considerable bias that goes into selection of what things to include tests for refusals on.

There's a difference between bias and area of focus. A math test that asks questions about trigonometry is not "biased" towards trigonometry, as compared to a math test that asks questions about probability.

Selecting topics that are frequently commonly censored in Chinese media is a reasonable area to focus on, because this is a model produced by a Chinese company. People are interested in whether the typical patterns of Chinese censorship are being applied to open source LLMs.

The bias is in which historical events it will refuse to speak about, and the excuse it gives.

Reply on news.ycombinator.com