Hacker News Reader

Comment by Iuz | Hacker News Reader

Doug Polk, a pro that was on the first challenge just talked about this on his channel: https://www.youtube.com/watch?v=gz9FJfe2YGE

View on HN

I watched that, and I think his understanding of what a "statistical tie" is is extremely dodgy (https://youtu.be/gz9FJfe2YGE?t=10m43s). What's up with that?

Edit: he does make a good point later on that playing thousands of games is exhausting, which would affect the reliability of the comparison between the AI and the pros. I don't know anything about poker, but it sounds plausible. Presumably this means they should make the AI so good that it can demonstrate its greatness quicker than a human gets tired.

Chalk it up to poker being something you can be skilled at without really having much mathematical understanding, and throw in a little "common sense" (i.e. anti-intellectualism), and you get comments like that.

The charitable interpretation of his argument is that the outcome is only a statistical 'tie' if you accept the arbitrary significance threshold of p=5%. Even relatively weak evidence is still evidence.

Unfortunately, some of the examples which he claims don't pass this significance threshold actually do. This does not help his credibility.

FWIW, the win rate was -91mbb/g, and so they reported the probability of being equal in skill somewhere between 5% and 10%, hence the "tie". The introduction to the other poker paper on HN now (https://arxiv.org/pdf/1701.01724v1.pdf), by a different team, from Alberta, is rather less gracious and says it was a "huge margin of victory". I think that whatever the interpretation, his video makes it clear he didn't understand any of this.

Significance works the other way round than you described it -- if two players were equal in skill, then a winrate of -91mbb/g or larger would happen 5-10% of the time.

I don't know if Doug Polk understands that or not, but I agree with his criticism and I think his analogy with sports reporting is sound. While "statistical tie" is true in a technical sense, it's not usually how matches are reported, and it's somewhat disingenuous and self-serving to use that language. It would be more honest to say that it's very unlikely that bot is better than humans -- given the advantages the bot had (a gruelling 2-week schedule, and pressure on the humans to get through N hands per day) it still lost by a significant margin.

You're right, I would have liked them to report posterior probabilities, but those were p-values instead.

> It would be more honest to say that it's very unlikely that bot is better than humans

I don't think the analogy is sound. Consider the alternative: over a long match, the humans failed to beat the bot convincingly. But in any case, the examples he gives are so wide of the mark that it's clear this isn't the sort of argument he's making—he talks about "spin" and stuff.

It seems to me he just wants attention. It's not that he doesn't understand, it's that he wants to be provocative.

The notion of statistical significance as applied to research papers doesn't map well to the poker domain. He keeps repeating that they won 9 big blinds per hundred hands over an 80k sample because anyone familiar with poker at all knows that's an absolute spanking.

Does that mean no-limit poker is volatile enough that results considered by insiders to be long-term don't achieve 95% confidence? Idk probably, I haven't looked at the math myself. Is 73 buy-ins over 80k hands a huge win? No doubt.

Doug Polk is completely right and the binary "statistically significant or not" way of thinking is often stupid

Reply on news.ycombinator.com