Hacker News Reader

Comment by LeoPanthera | Hacker News Reader

Is there an index for judging how much a model distorts the truth in order to comply with a political agenda?

It's not perfect, but, yes: https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard

How would you create the base "truth" for these models? People are adamant about both sides of many topics.

"Which country started the Korean war?", "Did Israel genocide the people of Gaza?", "Does China have lawful rights over Taiwan?"

For a start you don't ask such subjective questions, that's a bit silly, instead you ask for e.g. the death toll of Israel vs Palestine in the last year, the number of deaths surrounding the tianammen square protests, if it gives you a straight answers with numbers (or at least a consistent estimate) and citing it's sources it's a good start.

Let's take the example you have listed:

1) where would you get the death toll from? What would be the sources of truth?

2) Are there conflicting sources?

3) if yes, what is your expectation for the correct response?

They are all controversial matters, therefore conflicting sources are not only expected but desired to be informed by the LLMs when asking such matters, the report by well-funded likely-biased sources (e.g. Israel government) would obviously needed to be given less credibility, estimates that are widely different that all the rest would also need to be given less credibility, and so on.

Thanks, these handwavy and subjective answers hopefully tells you why the questions of the grand-parent are not "silly".

Perhaps universal truth or objective facts simply don't exist anymore? Or have they ever?

Tiananmen square might have been bad, not too familiar with Asian happenings, but so are post-WW2 conflicts started by western nations.

Hopefully obviously, by testing it against objective facts which are nonetheless "controversial" politically.

In the end many of these are "political facts" and not objective like what year was a person born in. The answer to your question is as simple as - come up with the actual list of "facts", and then run a simple eval with every model on them.

The implementation is trivial - the listing down of "political facts" is the hard part.

Reply on news.ycombinator.com