Hacker News Reader

Comment by malfist | Hacker News Reader

Maybe you should fact check your AI outputs more if you think it only hallucinates in niche topics

The accuracy is high enough that I don't have to fact check too often.

Without some exploratory fact checking how do you estimate how high the accuracy is and how often you should be fact checking to maintain a good understanding?

I did initial tests so that I don't have to do it anymore.

If there's one thing that's constant it's that these systems change.

Everyone else has done tests that indicate that you do.

And this is why you can't use personal anecdotes to settle questions of software performance.

Comment sections are never good at being accountable for how vibes-driven they are when selecting which anecdotes to prefer.

I totally get that you meant this in a nuanced way, but at face value it sort of reads like...

Joe Rogan has high enough accuracy that I don't have to fact check too often. Newsmax has high enough accuracy that I don't have to fact check too often, etc.

If you accept the output as accurate, why would fact checking even cross your mind?

Not a fan of that analogy.

There is no expectation (from a reasonable observer's POV) of a podcast host to be an expert at a very broad range of topics from science to business to art.

But there is one from LLMs, even just from the fact that AI companies diligently post various benchmarks including trivia on those topics.

Do you question everything your dad says?

If it's about classic American cars, no. Anything else, usually.

If you're not fact checking it how could you possibly know that?

Reply on news.ycombinator.com