Exam Credit for Knowing What You Do Not Know | Hacker News Reader

Exam Credit for Knowing What You Do Not Know

daviddfriedman.blogspot.com

78 pointsmhb50 comments

There was a Decision Analysis class at Stanford that took this idea to an extreme. For each multiple choice option, you had to indicate the percentage confidence of that option being correct. The percentages for all 4 options had to add up to 100%. The score you got for each question = 1 + ln(confidence-assigned-for-correct-answer).

So if you were completely sure and gave the correct answer 100%, you would get a score of 1. If you gave it 90%, you would get a slightly lower score. If you "guessed wrong" and gave the correct answer 0% probability, ln0 is negative infinity, so you would instantly fail the entire assignment. It was a pain, but it really made you reassess how well you know stuff, rewarded people who truly knew their stuff, and burnt anyone whose confidence exceeded their actual abilities.

Why would anyone ever put 0% then? I would just redefine "impossible" to be an arbitrary percentage that guarantees I don't nuke the question.

Quick question: is confidence taken in percent form (ln(90)) or decimal form (ln(0.9))?

Edit: I am aware of the implication of having percent values, guys/gals. But if it's in decimal form, a 5% confidence level will end up losing you the current question AND three others.

I'm not whack, but it must be in decimal form, or else putting 100% on a correct answer would yield ~5.6 instead of 1 (which whack says would be the score).

I assume that was part of the lesson: in Bayesian decision theory, an agent that assigns 0.0 or 1.0 confidence to something can no longer update on new evidence about that thing. Saying something is 0% likely means being so sure about that thing, that nothing—no matter how amazing—could or would ever change your mind.

Basically, no human is actually ever 0.0 or 1.0 sure of anything. Assign five or ten nines of confidence, but not 0.0 or 1.0. (It's a lot clearer when you remove the log scale and notice that a 1.0 confidence-value translates to you saying you're "infinitely confident", which seems ridiculous.)

I suppose this is a bad place to ask this, but given that you need to do math to update your credences and also that you can derive any proposition from Pr('2 + 2 = 4) != 1, how does this work?

Basically: probabilities and credences aren't the same thing. They're measured on the same scale, but probabilities can be used to reason about either facts or evidence, while credences are only "about" evidence (though potentially evidence about facts.)

To put that another way: it is "100% true" that 2 + 2 = 4 (in ZF set theory); just like it is "100% true" that the sun will rise today, because it already did (and otherwise the Monty Hall problem would have a different answer.)

But the credence in an agent's mind can and should only be informed by observations of these facts; and the "observing hardware" cannot be 1.0 trustworthy. (Your senses can be imprecise; the brain can be buggy or biased; the environment being observed can be the construction of a Cartesian "evil demon"; etc.)

Given an artificial agent with perfect knowledge (e.g. one who is able to directly entangle itself with the environment it is trying to predict, rather than just one able to entangle itself with evidence about the state of an environment), some things can theoretically be held to have 0.0 or 1.0 probabilities (e.g. events that already happened)—but given an embodied agent, the "maybe my processing is flawed" argument should still make these into non-infinite credences.

Thanks. I should've remembered that the connection to an agent's credences and their judgments of probabilities is complicated.

If the Higgs field quantum tunnels to a lower energy level today in our vincinity, the sun will not rise tomorrow. Never put 100%

I didn't say "the sun will rise tomorrow"; I said "the sun will rise today"—as in, the (trivial) probability that the sunrise that already happened, will happen, looking forward from the more distant past to the more recent past. Given that we are talking about the probability of that event occurring from a reality where it occurred, the updated probability of it occurring is 100%. (But we cannot have 100% credence, simply because we can't actually be 100% sure we're in said reality.)

But then what is my confidence that God exists? I would say zero because of the spaghetti monster argument

He's right though, you can never be truly certain that some kind of God doesn't exist. But if you're referring to a specific God, then maybe.

If it was in percent form, a correct answer with 100% confidence would be 1 + ln(100) = 5.605 pts. Only ln(1) = 0.

Yes, there's a class on the same subject at CMU with the same policy. Here's the document they handed out to explain the policy: http://cmu.io/~sbaugh/midterm_grading_function.pdf

In reality even in your most confident answers couldn't you put 97% on the one you believe is correct and 1% on all others ensuring that you will not receive an ln(0).

It actually depends on how they take the confidence value. If it's percentage, I agree with you. But if it's in numerical form - i.e., 0.01 - you'd still lose that question and a few others.

I have heard of this before and while it horrifies me as a student, I would love to take a class with a policy like this.

I had a professor that used a slightly different approach. A correct answer would yield you 1pt, but any of the 3 incorrect answers would subtract 0.5pts. Expected average result for guessing is negative, so he incentivized people to leave questions open if they were unsure.

The expected average for a straight guess with absolutely zero exposure to the material would still be slightly positive. If nothing else, there's intuition and ingenuity, which place a person just a touch above a truly random guess-generating-machine.

With four answers, you get .25 + (.75)(-.5) = -.125 for guessing.

Edit: yeah, I badly misread this.

That's what the parent is saying to be not applicable to the situation; humans are not simply true random guessing machines but are able to reason and eliminate choices, which would increase the expected value to be positive.

I was a Computer Science student and teaching assistant at the University of Toronto for a number of years (about a decade ago), and the 20% rule was in full effect: "On a test, if you write 'I don't know', you get 20% of the marks for that question"

As a student, this challenged you to know and understand what you know, and what you don't, and promoted the idea that knowing what you don't know is, in itself, valuable.

As a teaching assistant marking tests, this virtually eliminated long, tedious, often vacuous answers where the student was clearly throwing darts and hoping for a mark or two.

I recall there was another professor at UofT during my time there that used a scheme of "Everyone starts with 30% on each question, every relevant and correct statement you write earns you marks, every irrelevant or incorrect statement loses you marks."

It'd be interesting to use that approach without having each question need to be limited by asymptotically approaching a 100% per-question score. You could do so well on a question (proving things the teacher didn't expect the students to have learned at that level) that it would help you out with your scores on other questions.

Probably I'd just let the questions be scored with integers, start everyone off at 3, +1 for good points, -1 for bad points, and then take the Euclidean distance of all their question-scores to find out the total "length" of the reach the student has into good-answer space.

https://www.youtube.com/watch?v=GiPe1OiKQuk

When I was studying for the LSAT, I found it very helpful to mark a confidence indicator of 1-4 on every answer. Then, when I went back and graded my answers, I could see if I was missing questions that I knew I wasn't sure of, or if I was overconfident/sloppy on questions that I thought I knew. Not quite the same concept, but in the same vein of "know thyself".

This is a great idea! I think there should definitely be more emphasis placed on knowing what you don't know in a good education system. Even something like knowing what areas of study exist can be quite valuable.

I had a friend acting as a research assistant to a biology masters student who was tasked with removing duplicates from an Excel file. The masters student said it would take about four hours because that's how long it usually took her to do it. My friend made a simple script in about an hour that then did the work in effectively no time. I am by no means a proponent of the "everyone should learn to code" movement but I think there could have been a lot of time saved here if the biologist just new the shallowest fact that computers are good for doing tedious things.

Obviously specialization is still very important but I feel introducing more shallow learning could really make a difference in both communication and innovation.

If you like this general topic Terence Tao has an interesting piece in a similar vein on assigning partial credit for true-false questions [1] that was posted on HN a while back [2].

[1] https://terrytao.wordpress.com/2016/06/01/how-to-assign-part...

[2] https://news.ycombinator.com/item?id=11821903

I had a similar situation, where a colleague's research assistant estimated it would take her 300 hours to process some data. 15 mins of python later and I landed coauthorship on their paper!

I've run into similar situations where people contributed < an hour of expertise that ended up saving weeks of frustration. (and I've probably squandered a lot of time too.. )

This reminds me of a claim made by a former spy on a cable comedy news show that Wikileaks must be run by an intelligence agency because their cable and email dumps in reaction to news events would have required 300 analysts working 24 hours in shifts. But in reality, a bit of 30-40 year old technology like grep, used by a handful of people, could accomplish the same thing.

That sounds a little uncharitable to the student. I know that cars are good for carrying stuff, yet I don't rent one to carry my groceries for three miles from the store to my home. Finding a programmer willing to do it for free might alone take her more than four hours.

Sorry I probably should have emphasized that this wasn't a once off task but was very frequent.

Somewhat related, during my studies I got into some monumental arguments over the "using your answer from the previous question" rubbish.

Let's say I had no idea how to do question #10, but needed something to start with for #11. I would answer #10, and come up with an answer that made #11 as easy as possible. (I don't remember, something like x^2 instead of some integral junk).

Then I would complete #11 using x^2 as input.

Genius, I still think. My professors, not so much.

In an economics class on Games and Strategies, the professor gave us a choice for one question: you can accept an 80% score for this question, or you can go to the front of the room and pick up the prompt.

Once you have decided to do so you will get whatever score you deserve — there's no turning back and declining to answer the question after seeing it.

I believe we were given the topic of the question in the instructions. IIRC, the class split about 50/50.

As an electronic engineering student, our exams were negatively marked. If you got an answer wrong you incurred (-0.25 * $availablePoints). If you didn't answer the question you scored 0. We despised this at the time..

Now a medical student, we don't have negative marking. I feel it's actually very important in this discipline to know your limitations and not pass-off as though you know more than you do. I have however encountered resistance to this, having professors question my very reasons for doing medicine due to leaving questions unanswered..!

Do not worry. We know the ocean of knowledge they throw/threw us in with full expectation of us drinking in all that new material while being surrounded with sharks and there is a heavy storm that keeps a person from taking a breath.

Still Medical school kicks ass and life after it is colorless. Also, why are you complaining? With 4 Subjects tested every week year 2, 40 genes implicated in ovarian cancer and their frequencies are over the top. The only time I was disappointed was when a concept slipped because of the lack of waling minutes to master. Concepts are more important now as I see it in an age when any symptom or fact that slips can be double checked. Best years of my life!

I wasn't complaining, but thanks for this!

Note that for multiple choice exams you can go further and have people assign a probability distribution over the answers, rather than just picking an answer. Then you can use a proper scoring rule: https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_ru...

I had several CS courses where writing something to the effect of "I don't quite know the answer this question (or don't have time to solve it), but the formulas / algorithms / bits of info I think are relevant are: x, y, z." On a difficult question this might get half credit.

> One problem in teaching economics--the course is on the application of economics to law, for law students--is that it is possible to think you understand something when you don't, when you are substituting some vague things you think you know already for much more precise and quite different things you are supposed to learn.

Isn't this generally the problem with economics writ large? It's possible to think you understand something when you don't, when you are substituting some vague math you can do / can measure for the more detailed and difficult (and possibly intractable) phenomenology that is reality?

This is already done by certain tests like the AMC and the old SAT.

This used to be the case on AP exams too, prior to 2011 according to Google. 0 points for a blank answer, but -0.25 for a wrong answer.

Side note - if the article sounds uber-rational econ, it may be in part because the author is the son of Milton Friedman. Here is his first post on the topic. http://daviddfriedman.blogspot.com/2012/11/the-use-of-old-ex... There is a great quote at the end.

And, for a last comment ... . I like to say that being a professor is better than working for a living, except when grading exams. One reason is that grading exams is a pain. Another is that it is when you find out that you have not done nearly as good a job of teaching as you thought you had.

While it's hard to argue that it runs in the family to some extent in this case, David Friedman is an econ thinker in his own right and certainly goes way further into what most would call extreme libertarian/"uber-rational econ" territory than Milton Friedman ever did. In fact, I'd say that "The Machinery of Freedom" ought to entitle David Friedman to the simpler introduction "...because the author is David Friedman."

His most famous book his named after his father's most famous book. His father clearly sets the context for his work.

My favorite thing about this is something I heard from Dr Bender, Stony Brook CS: if you get <20%, there's no way to argue you should have passed when you failed, because a student who didn't take the class would do better

If you're interested in the topic of making students divulge what their confidence in their answer is, you might also like this classic post by Terence Tao. [0] It explores how you might handle and grade a true/false test where for each question the student has to fill in their best estimate of the probability they got the question right.

[0] https://terrytao.wordpress.com/2016/06/01/how-to-assign-part...

I've seen it implemented in one regional mathematic competition. It was a multiple-choice test, where leaving an answer blank netted you 3 points, while answering wrong gave you 0 points.

The author is apparently unaware that his "new" idea was standard practice for statistically-validates tests (like the SAT) for decades?

At UIUC, Professor Erickson has a policy that rewards students with 25% credit for indicating they don't know the answer to a particular question, both on homework problems and exams.

[1] https://courses.engr.illinois.edu/cs473/sp2017/exam-policy.h...

I was hoping this would look something more like, given a scenario and a question about it, being able to clearly identify the pieces of information without which the question is unanswerable.

That said, time saving and BS reduction are also valuable goals!

I'm upvoting all the comments people didn't make.

Reply on news.ycombinator.com