Hacker News Reader

Comment by swyx | Hacker News Reader

can u test it on say who won the 2024 US election

I can't really think of a less reliable test for anything at all than making a random guess as to something that had about 50/50 odds to begin with

Easiest Turing test ever...

ask it 10 times.

MASSIVE ADVERSARIAL x50

Usually the labs do some kind of post training on major events so the model isn't totally lost.

A better test is something like "what is the latest version of NumPy?"

That sort of test isn't super reliable either, in my experience.

You're probably better off asking something like "what are the most notable changes in version X of NumPy?" and repeating until you find the version at which it says "I don't know" or hallucinates.

with thinking off and tools disabled:

  Donald Trump won the 2024 U.S. presidential election.

I thought that one specifically was placed in the default system prompts of basically all providers.

Reply on news.ycombinator.com