What do you use as “ground truth”? The page says “independent sources”, and I’m sure there’s too many to list, but my question is how are they vetted as being truthful and how are two sources with opposite viewpoints reconciled?
Will read reliable-sources page properly.
It can be, but that's hard to correctly judge on at scale.
What's dangerous is treating information from an anonymous author as factual. You gotta check that in other ways.
Detailed report, if you are interested: https://github.com/SerhiiKorniienko/bullshit-detector/blob/m...
I imagine this is similar to how Twitter's/X's community notes work. Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true.
So from that approach, you actually want to have two AIs that are on the other side of the spectrum of whatever you're trying to find out if it's true e.g. if a conservative AI and a liberal AI both agree that something is false, it's highly likely to be false.
I don't think LLMs can discern truth when the majority of news sources are incentivized primarily for views.
> Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true.
Outfits that blindly parrot talking points would not be at odds and thus not be good candidates.
Sounds challenging to me...
I'll work on empirical sources improvement in a next releases.