Hacker News Reader

> As you may see, the scores are around the random guessing level, with the smallest model being below that threshold.

Err… can someone explain to me what is meant here? Surely the model wouldn’t consistently “guess wrong” compared to randomly, as that would be better. I guess some things like general coherency (i.e. is it even readable or gibberish) factor into that score?

View on HN

Those are multiple-choice questions. If some of them are "trick questions", where obvious answer (e.g. the value taken directly from question's text) is wrong, bad model might perform worse than a dice.

On the other hand, not sure where from 25% baseline for random answers come from. Since this is multiple-choice-out-of-4 test, random guessing should be correct in 1 in 15 cases, not 1 in 4.

Reply on news.ycombinator.com