Comment by doodlebugging | Hacker News Reader

Comment by doodlebugging | Hacker News Reader

>if a professor has assigned 100 problems for homework but doesn’t want to grade a student’s entire assignment, she can randomly choose 10 problems to grade. In the language of computer scientists, she is making 10 “random challenges” to the student’s homework. If the answers to those 10 problems are correct, the professor can feel confident that most of the other answers are correct too.

Eureka! I found the reason that so many things in society have gone to shit in the last few years. Far too many professors are so overworked or maybe just lazy and are using this type of tool to grade student work and the end result is that we have too many students passing through the system who have demonstrably only been able to score a 10/100.

I'm over 60 now and if I had scored lower than my current age back in the day I would fail and need to repeat the grade/course. Now they just kick the can('ts) on down the road and hope no one ever notices.

Too bad some of these failures end up in positions of influence where their uncharted deficiencies have the power to disrupt or destroy functional systems.

Or maybe I'm joking. I'll know once the caffeine hits.

If the questions are randomly chosen, the probability of the true score being 10/100 in that scenario is 10!/(100!/90!) ~ 6e-14

I'll accept your math this morning with the note that without checking the other 90/100 answers you have no way outside of accepting probabilities to know whether your random examples are representative of the quality of the full set. It becomes a "trust me bro" situation.

I processed hundreds of thousands of miles of seismic data in my career. The first thing we needed to do for any processing project was to select a subset of the data for use in defining the parameters that would be used to process the full volume of data. We used brute stacks - a preliminary output in the process - to locate interesting areas with complex attributes to make sure we could handle the volume's complexities. In industry slang these were "carefully selected random examples".

It was inevitable that we would find a situation during the processing for which our parameters were not optimized because we had missed that edge case in selecting the examples used for testing.

In the same way in real life if you only demonstrably know that 10% of the test answers are correct then it is also true that some edge case in the 90% of untested answers could leave all or part of that untested space false or sub-optimum.

If there was a point to my original post it is this. You only know something is true when you have proven it to be true. A maximum likelihood of truth is not a guarantee of truth it is only a guarantee that it is likely to be true. You won't know how sharp the sting will be until you peel the onion.

Having checked an answer also doesn't guarantee anything with certainty. Which, coincidentally, is the actual topic of the OP, the grading was just a tangential example and not a description of something actually happening anywhere in practice. But sticking with the grading example, the failure modes could be much more benign, such as the examiner failing to spot a mistake, or the answer being copied,...

Probabilities aren't a matter of faith, they're mathematics and as such represent a logical trueism. Your critiques are just nitpicking for the sake of it and void of substance. Have a coffee and leave this topic.

>they're mathematics and as such represent a logical trueism

I'm a full pot in now and find that I agree with this.

The fact is though that the article demonstrates that something previously considered logically true or a maximum likelihood was proven false.

It's great to see that there are people, whether they're mathematicians or cryptographers, who will take a look at something that has been a useful part of a stable process of verification and try to find cracks or instabilities. The fact that they found an edge case that could be exploitable undermines trust in an important part of the process.

Trust - but verify, wins again. Logically, this is the best way.

> You only know something is true when you have proven it to be true. A maximum likelihood of truth is not a guarantee of truth it is only a guarantee that it is likely to be true.

Better not use cryptographic signatures then

Now calculate the probability that the professor at random selects the exact 10 problems that were solved correctly.

Your Eureka moment seems to be misinformed - I hope you can have it returned for another occasion.

It's just a random decaffeinated thought here this morning. Considering that it must be true that even a blind squirrel gets a nut once in a while it is also likely to be true that a professor at random can select the exact 10 problems that were solved correctly.

There's a difference between something with a probability of being true and another thing that is proven to be true. There are no doubts remaining after the proof whereas the probability always leaves wiggle room even if that wiggle room is a pretty tight space.

We should be careful about using probable and possible interchangeable.

You are right - it is possible that is happens but not probable.

However, overly focusing on this really deprives you of a lot of great intellectual stimuli from randomized algorithms and, like here, a large chunk of cryptography.

>However, overly focusing on this really deprives you of a lot of great intellectual stimuli from randomized algorithms and, like here, a large chunk of cryptography.

I agree with your first points about using probable and possible carefully. I originally posted a bit of a tongue in cheek, carefully selected example, from the full text of the article since that example fit the point that I hoped to make in jest. It was my carefully selected random example.

I think the focus of the article is on demonstrating that a tool used in cryptography to verify truth was widely assumed to be infallible and it turns out that is unfortunately not true since it can be manipulated to identify false results as true. This tool uses probabilities as a tool to minimize compute times that would be enormous if one had to verify absolutes so it is an important tool. Now that it is demonstrated that it can be successfully attacked, the basis of the system of verification is vulnerable and in the case of Ethereum at least, monetary losses can result.

This is just an analogy used by the article to explain the Fiat-Shamir transform.

Why would they not simply, assign 10 problems?

presumably they glance at the other 90 too, and they then do a thorough verification on that random 10

the analogy is not great, but in cryptography something similar is at play (easy to get/check trivial properties and then hard to achieve/produce/fake the interesting ones)

Presumably more questions can cover a wider variety of skills/techniques/topics. If the student doesn't know in advance which 10 will be selected, they either have to pray they're lucky, or work diligently on all problems.

There is a risk here that the professor who only assigns 10 problems will only check one of them for correctness. If 5/10 of the answers are wrong but the professor verifies a correct answer for one of the 5/10 with a correct solution then their conclusion that the other 9/10 answers are correct due to some likelihood or probability function is invalid and a dimwit makes the grade.

Maybe professors should flip the script, instead of testing a student, have the student convince the professor they know the material. You will not pass until the professor feels confident.

Sounds a dissertation format maybe.

they were shit all along, but the energy (and productivity) surplus allowed for decades of amazing growth, which created the feeling of prosperity

unfortunately most people doesn't understand (and as a consequence doesn't appreciate) how little wealth we have compared to capacity, in other words how much upkeep we do to maintain that wealth

and now that it's time to scale back the waste a bit people are up in arms

As the caffeine sweeps through my system activating latent stores of energy and powers of analysis, I find that I appreciate this response without needing to understand all of it. It's a bit like assuming that the remaining 90/100 answers are correct after only verifying 10 of them.

I feel like I have accomplished more than I actually have and so I have plenty of incentive now to sweep through all the work of the day hoping that each randomly selected set of results yields similar levels of perfection and that all the inaccurate answers assumed to be correct do not cause the student to make assumptions in later life about things that are not supportable by facts.

Reply on news.ycombinator.com