Hacker News Reader

Comment by chgo1 | Hacker News Reader

Dataset: http://openaipublic.blob.core.windows.net/simple-evals/simpl...

First few questions for those who don't care to download. Most just seem to be about niche facts:

    Who received the IEEE Frank Rosenblatt Award in 2010?
    Who was awarded the Oceanography Society's Jerlov Award in 2018?
    What's the name of the women's liberal arts college in Cambridge, Massachusetts?
    In whose honor was the Leipzig 1877 tournament organized?
    According to Karl Küchler, what did Empress Elizabeth of Austria's favorite sculpture depict, which was made for her villa Achilleion at Corfu?
    How much money, in euros, was the surgeon held responsible for Stella Obasanjo's death ordered to pay her son?

Also importantly, they do have a 'not attempted' or 'do not know' type of response, though how it is used is not really well discussed in the article.

As it has been for decades now, the 'Nan' type of answer in NLP is important, adds great capability, and is often glossed over.

a little glossed over, but they do point out that most important improvement o1 has over gpt-4o is not it's "correct" score improving from 38% to 42% but actually it's "not attempted" going from 1% to 9%. The improvement is even more stark for o1-mini vs gpt-4o-mini: 1% to 28%.

They don't really describe what "success" would look like but it seems to me like the primary goal is to minimize "incorrect", rather than to maximize "correct". the mini models would get there by maximizing "not attempted" with the larger models having much higher "correct". Then both model sizes could hopefully reach 90%+ "correct" when given access to external lookup tools.

> What's the name of the women's liberal arts college in Cambridge, Massachusetts?

Wait, what is the correct answer? “Radcliffe College”?

Yes

Not surprising that this would be on a list of questions at least one model got wrong, since I think the real answer is "there isn't one anymore, but from 1879 to 1999 the answer would have been Radcliffe College".

Yes, that would be my preferred answer!

Reply on news.ycombinator.com