Hacker News Reader

Comment by ben_w | Hacker News Reader

On the topic of trusting surprising observations, I remember a quiz at school where almost everyone had the same failure mode.

First question: "read all the questions before answering"

Questions 2-99: Trivial but interestingly bizarre questions, like "draw a triangle with blue ink"

Question 100: "Now you have read all the questions, answer only questions 1 and 2"

I was very tempted to make the mistake everyone else made, and quite smug when I finished, sitting and watching as everyone else asked for ink and later on ink erasers.

View on HN

I've noticed that most people don't seem to look at things like this from a meta level.

I was given that same quiz at school when I was about 12, and I remember I skipped to the end and read the last question (which reveals the trick) because it was highly suspicious that we were randomly given a quiz with a whole series of questions that began by telling you to read the whole thing.

Similarly, often quizzes/surveys/studies have an obvious answer when you include the meta-fact that the question is even in there. For example maybe you're shown an image like this[1] with the question "which glass do you think holds more liquid?" Well, it looks like the tall one, but that means it's probably the short one.

But we had a good percent the class going for ages on that "read all questions" quiz.

[1]https://i.imgur.com/6xkc813.jpg

In my day job, I deal with the development of, and verifying the usefulness of, tests.

The lesson here is that most tests are designed poorly. Most people don’t realise the effort that goes into doing it properly. Every item needs to be proven to be useful. Items that either purposely or accidentally ‘trick’ the test taker often prove to not be very useful when you run the stats. If a test ‘breaks the fourth wall’ like this, the item probably won’t stand on its own to be useful.

Ah, that explains something I have been confused by for a while now without really understanding why.

I'm learning German, and one of the apps I'm using is Clozemaster, where a sentence is missing a word and I have to fill in the gap. The easy mode is to choose from a list of four, the hard mode is to type in the word. Easy mode I can do fairly reliably without even fully reading the sentences, hard mode I struggle with — while the latter is partly because it only accepts one single specific word (so if the sentence would be valid with another word, e.g. weiß/blau/gold/schwarz, keine Punkte für mich if I pick the wrong correct answer), that alone didn't feel sufficient given the subjective experience of the difference between them.

I think, from what you've said, in the easy mode I'm probably picking up on things that aren't as useful as I need.

Thank you. I've seen surveys where every question is a 'trick' question! Where the correct answer is always the non-intuitive one. Usually when the surveyor is trying put push an agenda and prove that you're wrong about something, but it just ends up being patronising if you realise the trick.

In my experience, the actual trick when presented with a pair of glasses like that is to determine whether the answer is "the short one" or "neither, they hold exactly the same".

Haha, very true.

There’s a problem with this quiz. why would question 100 have the special privilege of discarding all the other questions ?

From a purely linguistic pov, this question 100 isn’t a question, it’s a meta-information about how you should reply to the quiz itself. I hate it when teachers try to be too smart for their own good.

It's a simple case of teaching the importance (according to the teacher) of reading the instructions before starting the work. It's not that deep.

It's a simple case of willfully hiding important information in the last place anyone will look. Awareness of these tactics is a useful skill to have in low-trust societies, and when used as a teaching tool it will guarantee the outcome to be a low-trust society.

The world is messy and "the rules" aren't often written out for us in black and white. Teaching somebody to deal with uncertainty by way of studying the problem before jumping in without the full picture doesn't seem like it would create a "low trust society" as you claim. I would suggest it creates people who might look before they leap.

We have enough people on this planet going through life without ever once thinking about what they are about to do.

I had a teacher that did this but his whole MO was teaching us good test taking habits. You speed through the easy questions so you know how much time you have to devote to the hard ones. I.e., if that’s the last place you look, that’s an issue.

As a student, i remember very vividly that any potential imprecision in the question of the exam was the source of great confusion, especially since you couldn't ask the examinator for explanation (the test was supposed to be perfect).

I'm not being pedantic here, i am really furious against teachers that makes students think about the best way to disambiguate the question instead of thinking about the subject of the exam itself.

Typically the exam had pointless questions and its only purpose was to teach this particular lesson. I've never seen nor heard of a teacher using this on a test that actually mattered.

> meta-information about how you should reply to the quiz itself.

This is referred to as "rubric", and paying attention to the rubric on exams is a very important skill. Especially when some of them are confusing, and you get "answer ONE question from section A and TWO questions from section B BUT no more than ONE question marked with a (+) in total" on a paper that folds out to thirty questions.

While it's often infuriating it's useful to know how to operate in an environment where the rules will be read pedantically against you.

A rubric should be specified outside of the context of a specific question if it applies all questions.

These are all very cute taxonomical arguments that a) ignore the point of the exercise and b) are only interesting to software developers.

> This is referred to as "rubric", and paying attention to the rubric on exams is a very important skill.

Following arbitrary and pointless rules and playing mind games with the test giver. Only in the made up world of academic testing.

And taxes.

Oh thank you, I needed that laugh.

All organisations have arbitrary and pointless rules. It ensures that people know their place.

It'd be better to name them "Task 1" etc, instead of "Question 1" etc. Especially considering they are not really questions but tasks (i.e. "read all questions.", "do question 1 and 2", "draw <whatever>", etc).

Given this was at least 25 years ago, they might well have been. It's not like human long-term memories are vastly more reliable than things like Stable Diffusion.

On a similar level, from a purely linguistic point of view the initial instruction should be "read all the questions before answering any of them". Any pedant could have great fun arguing that reading a question before answering it for the set of all the questions did fulfil the brief of "read[ing] all the questions before answering them"...

Right. Kids who answer the “wrong way“ from top to bottom got answers 1 and 100 incorrect, but correctly answered questions 2 through 99 and score 98/100.

Kids who read the last question and leave the rest blank get two questions correct for a score of 2/100.

On a typical test the blast radius for a question is limited. There is no convention that getting a question wrong alters the outcome for other questions, too.

One of my kids had the same sort of quiz/test given, but he and his friend had a concern with it. When I read it, I was inclined to agree with them that it doesn't work the way the writer intends.

Presumably, each question contains an instruction to follow. Being instructed to read all questions first doesn't mean you then do what any of the other questions instruct. Why, after reading all other 99 questions first, do you do what question 100 instructs but not what the others instruct?

Because you get in the mind of the test-giver and correctly predict the point they're trying to make and act accordingly. If you can't or won't, you fail. That's what school is about, after all.

Can't let these natural intelligences loose in the world without first aligning them.

Sadly true, which leads to a conundrum when trying to get inside the test-giver's head: did the test-giver notice this intricacy of the question and therefore I should give the clever answer, or did they not realise and therefore I should give the naive answer?

In this particular instance, my son and his friend were unable to convince their teacher that the test/quiz was flawed, so the naive answer was the "right" one to win the teacher's approval.

I would have answered wrong even if i read all the questions beforehand. Answering 2-99 gives you a score of 98/100, while doing it correctly gets you 3/100.

You can argue that you read the questions first too, so you should give yourself the answer to querstion 1 as correct as well and have 99/100 overall

I always found this kind of "test" infuriating.

I just answered all the questions out of spite.

So you interpreted in a way, where you executed question 1, and then question 100? That seems counter-intuitive. Even with the knowledge of the questions, there is a case that your should still execute them in order.

I think this test is pretty ambiguous.

If you answer all the questions after reading all of them, you could only get only question 100 wrong and get the other 99 correct, so reach a score of 99/100.

If you answer only questions 1 and 2, you get the answer to 100 right too, so you could score 3/100.

In the latter case, you could score 3/3 too, depending on how the examiner decides to mark it.

So which is it?

This is part of why I think it's highly relevant to the AI/LLM discussion.

Now sure, with the benefit of 25-ish years of hindsight, I think that while my actions were correct in the situation and the point of the exercise was to get the other kids to think more like I already did, the "strictly correct" response — and what we would demand from an AI for safety reasons right before complaining about how weirdly inhuman this is — would be to answer Q1-Q99, then for Q100 write down the answers to Q1/Q2 again (and as Q1 was just "read" that just meant Q2).

> First question: "read all the questions before answering"

But that's not a question.

Reply on news.ycombinator.com