Confidence Scores for Exam Questions | Hacker News Reader

Confidence Scores for Exam Questions

nomagicpill.substack.com

14 pointssurprisetalk12 comments

I seem to remember some medical related multiple choice tests in the UK use a mechanism of +1 for correct , 0 for unanswered , -1 for incorrect.

A system like that seems especially appropriate for a practice where the foundational principle is "do no harm."

Would probably be applicable to engineers as well, or any other field where the practitioner has an obligation to be aware of the limits of their competency.

and yet.

Typically you have -1/N for incorrect selection, where N is the number of choices. For N=4, you would grade incorrect answers as -0.25.

If you have a person taking an exam that is not confident in themselves or generally knows the subject area, you don't want to negatively impact educated guessing.

> Typically you have -1/N for incorrect selection, where N is the number of choices. For N=4, you would grade incorrect answers as -0.25.

That is definitely not typical. -0.25 is the appropriate adjustment for N=5. For N=4 you want -0.33. -1/N makes no sense at all.

Note that doing this preserves the expected value of everyone's score, but artificially widens the variation, which you might not want. It does allow you to diagnose partial knowledge, which you probably do want.

Maybe it is typical only for me. I did question it and was told that we don't want to completely remove the incentive for educated guessing. We also usually have a scale of question difficulty, so getting people to a pass is not too difficult if they know the subject at all, but getting towards 100% gets significantly harder.

I think the real reason is that our questions are usually N=4, negative marks of 0.25 allows for quick adding.

> I did question it and was told that we don't want to completely remove the incentive for educated guessing.

So... you were told some unmotivated nonsense?

On an item with four answers, +1 for a correct answer and -0.25 for a wrong answer means that in expectation you will receive 0.0625 points for a completely uneducated guess. The only correct adjustment you can make is to dock 0.33(3...) for a wrong answer, in which case an uneducated guess is worth 0.0000 points and a minimally-educated guess, one in which you're capable of eliminating just one of the four answers, is worth... 0.0833(3...) points.

> I think the real reason is that our questions are usually N=4, negative marks of 0.25 allows for quick adding.

You think adding fourths is easy, but adding thirds is hard? If you really believe that, it'd be simple enough to add fifth choices to your questions.

Are you sure the real reason isn't just that nobody ever bothered to put any thought into what they were doing?

It would make more sense to just use IRT for grading the responses than trying to add more complexity to the answers themselves.

https://en.wikipedia.org/wiki/Calibration_(statistics)

https://en.wikipedia.org/wiki/Scoring_rule

Terence tao had a blogpost about this for the case of true/false questions, with multiple choice left as an exercise for the reader.

https://terrytao.wordpress.com/2016/06/01/how-to-assign-part...

At least with the formulation Tao had, one unfortunate side-effect is that if you answer with 100% confidence and get it wrong, you basically fail the entire course.

Moodle has had certainty based marking for at least 15 years.

https://docs.moodle.org/502/en/Using_certainty-based_marking

Reply on news.ycombinator.com