Hacker News Reader

Comment by impossiblefork | Hacker News Reader

I like that test where some of the questions are wrong and wonder whether we should have that kind of thing in maths textbooks.

I think people need to be trained to be more confident in what they know, and if we gave them that kind of thing we could maybe train them to become so.

View on HN

"incomplete information" is a standard concept in word problem curriculum. But usually it's explicitly an option in the test, as a fairness to the student.

Making mistakes in lecture is a standard technique used by good teachers, to promote active listening and critical thinking.

Yes, but in Swedish school maths books there are lists of problems, and these are always correct. You don't find a "this problem is nonsense, and here's way" in the list of solutions.

I think this kind of thing is extremely unusual.

I don't know about Sweden specifically, but I'm pretty sure that many word-problems-heavy education systems do also demonstrate ill-posed problems and expect pupils to recognize them. Of course there is generally a warning that some of the problems you get may be ill-posed, and what to say in that case: it's not a complete surprise.

Actually - do they do this in LLM benchmarks? As a measure of overconfidence/confabulation? Seems immediately applicable.

I don't think it's a common thing in any public LLM benchmarks or in any standard QA datasets. Maybe in internal stuff at AI firms.

I didn't see that in the document. What page is it on?

I think they mean at the bottom of p216 (pdf page 4), where he says he doesn't know, r+s=80 but there isn't enough information to solve for r and s.

There's two questions that are intended to be wrong (probably to test confidence). One with insufficient information and where the question itself implies falsehoods.

The questions are on page 215 (3/26) and Tao's answers are on the next page.

Reply on news.ycombinator.com