Hacker News Reader

Comment by Tinyyy | Hacker News Reader

This is an interesting article, but there's one part that I don't really understand: 'After scaling up the approach by increasing the dataset size, we de-anonymize 600 programmers with 52% accuracy.'.

Isn't 52% close to a random guess? I don't get how this ties in with the rest of the paragraph either.

View on HN

If they were saying that in 52% of the cases, they could guess the correct choice out of 600, then random chance alone would have been <.2%.

If they were saying that in 52% of cases, they could guess if some code did or did not belong to a given programmer, then it is basically a random guess.

> If they were saying that in 52% of cases, they could guess if some code did or did not belong to a given programmer, then it is basically a random guess.

That entirely depends on the distribution of the original sample.

Does it? If you're asked "does this code belong to this programmer?" and flip a coin, you'll approximate 50% regardless of distribution of programmers.

Of course, "const No" might do quite a bit better than 50%, depending on the distribution of the original sample (for that matter, so might "const Yes", but those distributions seem less likely).

Well, if there were 600 programmers, one of whom had written 649 programs and 599 of whom had written 1 program each, you could achieve 52% accuracy by always guessing the same guy.

Of course, the paper explicitly says "For this experiment, we use 600 contestants from [Google Code Jam] with 9 files" so I think in this case the distribution was probably fairly even?

I think you missed some context. Up-thread, someone raised the point that it's unclear whether the question was "Of these 600 programmers, which wrote this code?" or "Did this programmer write this code?", and this subthread is discussing the latter case.

Yeah, I had an underlying assumption in my mental model that they would have split the questions so that half were yes and half were no. But you are right that that assumption may not hold. A random programmer with a random program will be a 'no' far more than 50% of the time.

Considering there are 600 programmers, 0.166% (1/600) would be random guess.

No, a random guess would give you 0.16% accuracy.

52% is close to a random guess, yes: between two alternative outcomes, such as heads or tails in a coin toss.

If the question is, which of these 20 programmers wrote this code", then a random guess has only a 5% chance (1/20) of being right! So 52% is more than 10X better than random.

Reply on news.ycombinator.com