Hacker News Reader

Comment by azakai | Hacker News Reader

The claim that it's statistically unlikely to get those random numbers isn't entirely clear. When asked how unlikely it is, the response is "extremely unlikely", but it's important to quantify that. It's possible he's playing 20x more games than them, for example, and then we'd expect his best scores to contain more unlikely results. But it's hard to say if they would be as unlikely as what we see.

View on HN

> It's possible he's playing 20x more games than them

Possible but unlikely b/c the person making this claim is the current No. 2 Donkey Kong player in the world Wes Copeland, not Joe Schmo off the street.

Even if Mitchell's disputed scores were accepted they would still be far off Copeland's scores. So when the No. 2 player speaks it carries more weight than just some random dude in his basement.

Wouldn’t a pro player be playing less games? They’re playing longer, higher scoring games not lots of average scoring games.

Sort of. However, at that level, if you're in the middle of a game and aren't on track to getting a high score, you're probably going to abandon the game and try again from the beginning. Therefore there'll be a fairly high number of games, a sizeable proportion of which are abandoned.

It's like when speedrunners attempt to complete games in the fastest time. If they make a mistake at any point in the game that will cost them a lot of time, they will most likely abandon the game and keep trying until they are able to complete the game without making any costly mistakes.

It depends how much time they put into it. No one checks how many games they play, so one player could play only on weekends while another might also play a few hours every evening.

Unless the rng is really broken and unstable, there's no reason to get a different result in any top run of the game. The longer you play, the closer you should be to the average.

What? Of course there's a reason! It's because you only post top runs! That's like saying "there's no reason lottery winners will have hit more winning numbers than anyone else", but they're lottery winners exactly because they did!

Exactly. If a game has an element of chance, then a player's top scores will tend to be top scores in part because of luck helping there (same reason as why regression toward the mean [1] exists).

In such a game if each player can choose how many times they play then they can gain an advantage simply by playing more. And their top games will look more and more statistically unlikely. That can be especially surprising if they only report those top games, which is the case here.

What would be interesting to compute is how many more games does a player need to be playing in order to reasonably get the results in the article. If it's 2x, it's not proof of cheating. But if it's 1,000x then maybe it is, because who has that much free time?

[1] https://en.wikipedia.org/wiki/Regression_toward_the_mean

That's assuming that the random number generator is truly random, and not a deterministic pseudo-random-number-generator, which will _necessarily_ produce a sequence of numbers that satisfy certain statistical properties, over a long enough run.

It's unlikely, but possible, to roll a 6-sided die 100 times in a row and get 100 6's.

In basically every PRNG algorithm, there is _no_ reachable state which will result in 100 6's in a row.

Are you sure about this? The period of a Mersenne twister is 2^19937 - 1. I'm pretty sure somewhere along that inconceivably large number of states we can find 100 6s in a row.

That's only about 6000 decimal digits long - from a quick visual check the longest run was four sixes in sequence:

http://lcn2.github.io/mersenne-english-name/m19937/prime-c.h...

Expect to get length 1 in a number 10 digits long, 2 in a number 100 digits long (10²), 3 in a number 1000 digits long (10^1000)... So expect to find a run of 100 6s in a random number 10^googol digits long I think.

No, the RNG doesn't output successive digits of the prime. That would be period 19937. It's period 2^19937. The random number is 10^6000 digits long.

> 10 digits long[...]100 digits long[...]1000 digits long[...]10^googol digits long I think.

Not 10^googol digits. Googol digits long. Except you don't even need that much, because die rolling is base 6.

> It's unlikely, but possible, to roll a 6-sided die 100 times in a row and get 100 6's.

It's exactly as unlikely, but possible, to do the same and get the exact sequence: 6 2 2 4 5 4 4 1 4 4 4 3 5 4 2 4 6 2 4 4 1 1 2 4 4 4 6 6 4 6 6 1 1 2 4 4 1 5 1 2 3 6 3 3 3 1 6 4 1 4 1 5 4 2 2 5 6 2 1 6 3 5 3 3 2 6 6 6 3 6 5 5 1 5 4 6 2 6 4 3 2 2 5 1 5 3 3 5 4 5 6 4 1 5 5 6 6 6 6 3

(generated via https://www.random.org/integers/?num=100&min=1&max=6&col=5&b...)

The longer each game session goes, the closer the RNG will be to average. This will happen even if you're only posting one-in-a-million runs. One-in-a-million when you pick 10 numbers is extremely biased. One-in-a-million when you pick a thousand numbers is only subtly biased.

Yes, the longer the game is the closer the average will be to the true average. But given a fixed or bounded game length, you can play enough games to get as much deviation from the average as you want.

Again, the numbers need to be run. There's no way to tell without them, the truth could be either way.

> But given a fixed or bounded game length, you can play enough games to get as much deviation from the average as you want.

If "you" is some abstract entity with infinite time, perhaps. Even if you do ten three-hour runs a day, you can't actually get a very high deviation from average in any given century. Getting a certain deviation requires exponential time in proportion to the number of RNG calls.

You're assuming those are random numbers. As the article says, they are only semi-random (that's one of the big surprises they found).

And you can see from the numbers that it's not just Mitchell's values that deviate from the expected mean, it's the others as well (he's just more extreme), which also shows we can't expect them all to converge to the mean, there's some more complex statistical process going on.

His deviations being more extreme might be explained by him playing more games, or it might be cheating. So far we can't tell.

Yet every run still has many thousands of RNG invocations for that particular metric. To be off by a few percent, safe to say that RNG would fail any randomness test.

Are there thousands of smashes per game? (I'm not a Donkey Kong player.)

Yes, the RNG is not perfect, as the article says,

> the points derived from those enemy smashes are assigned semi-randomly

(which apparently solves an old mystery there).

In any case, the numbers for total smashes is 380 for Mitchell and 242-371 for the other players. Percent of score from smashes is 17.7 versus 10.6-14.8. In both cases the large variance among the other players shows that while he's an outlier, we can't expect the numbers to be close to a specific average, this isn't a simple statistical process where we average out thousands of independent variables.

This is easy to resolve. Just dump the ROM and look at the RNG.

The full details are here: http://www.classicarcadegaming.com/forums/index.php?topic=41...

But what Wes showed was that all the top gamers (with higher scores than Billy) average 13-15% of their total points from barrel smashes while Billy averaged almost 18%.

Details: https://pbs.twimg.com/media/DVDyEHrX4AEnxP0.jpg:large

While that thread breaks down the idealized probabilities, it doesn't say anything about the RNG behavior itself. Some generators have bad seeds. It's conceivable that Billy got lucky and hit one of these. I sincerely doubt that's the case. With the source it'd we could definitively confirm or rule this out.

Another possibility is that Billy figured out a hitherto unknown RNG manipulation trick. This is how a lot of TAS runs work. Again, I doubt this is the case but with the source it'd be easy to rule out.

I don't get it. "Percentage of points coming from smashes" seems like a worthless metric for detecting cheating via favorable RNG seeds. The relevant metric would instead be average points per smash. And for that Mitchell's results are not particularly out of line with the other entries (the distribution is 464-511, with an average of 480; 491 doesn't look at all suspicious in that context).

Wouldn't this rather be a sign of people making different strategic tradeoffs? Mitchell spent more time breaking barrels but at the cost of getting fewer bonus points for finishing the level in time.

Reply on news.ycombinator.com