Seems like having tests would've easily solved this issue.
The article leads with a clickbait question.
The reason they didn't notice is because the stats aren't readily available and public to everyone, to see the sharp uptick in wins paired with the game being filled with opaque systems. No player playing for the first time would notice or have a chance to notice.
To put it another way, this is an article about a (secret and mistaken) massive change to a very difficult game that made it slightly less difficult. Same article turns around and asks why nobody noticed, like that's an interesting question.
The stats have always been public and instantly available, impressively so. Here's the bot command to query the stats in question:
!lg * start>"2015-03-06 00:00:00" start<"2015-03-21 00:00:00" cv=0.16 / won
1180/39437 games for * (start>'2015-03-06 00:00:00' start<'2015-03-21 00:00:00' cv=0.16): N=1180/39437 (2.99%)
Using another query to determine the all-time winrate across all versions:
!lg * / won
The listgame interface is documented at https://github.com/crawl/sequell/blob/master/docs/listgame.m... , and the bots can be found on IRC and Discord.
No, lots of players noticed. The question is phrased as "how did the players miss this?", but that's not what the author means - what he means is "why didn't the community come to a consensus on what had happened within two weeks of the introduction of the change?"
Which is a much stupider question.
I don't think it's a stupid question at all; not when the effect is so large. It was literally double what it should be, leading to a 3x overall win rate.
Coming from Dota 2, if there was a bug like this it would be caught within minutes, if not seconds, and patched within an hour or two.
- Yes, the Dota player base is orders of magnitude larger, and the damage numbers are easily available. Also I have never played Crawl.
The article also brings evidence that many players didn't notice or believe that anything changed, they specifically believed that nothing did change, which is a surprising belief for such a huge change - but understandable in the context of of such an opaque system. I also believe that other players noticed something had made the game easier, but no one realized what specifically had changed, or most likely suspected how big of a change it was.
Not really. The article cites responses in online discussion to that effect. But there is no evidence that those responses came from someone who had ever played the altered version of the game. With certainty, many of them didn't. You don't have to do a playthrough on the latest version before leaving a new comment in a forum thread.
It's a safe bet that the majority of the player base never encountered the bug at all, since it was only available if you downloaded the game within a two-week window.
How would someone design tests that work well for roguelikes? It's not simply Human strikes Orc for 10 damage. Damage range may be a bell curve of 5-15. There's misses and hits. There's little bonuses that increase the miss probability, probably to ridiculous levels. There's armor, calculated from things like race, constitution, magic, divine bonuses, class mastery bonus.
And then you have procedurally generated... stuff. Equipment are the easiest to control. Some games have procedurally generated monsters, some have procedurally generated deities. To be able to reduce this stuff to tests kind of defeats the purpose of doing it, which is to be unpredictable.
Sometimes you have a combo of things that add up to 40% damage resistance. Is 40% too much or too little?
After all this, how do you detect that the bugged spear is doing an unreasonable amount of damage?
Wouldn't detect extremely improbable things, though, like having a spear with a super-rare effect + having armour with another super-rare effect, since those would show up so infrequently that they wouldn't skew the overall stats much.
and/or
(b) Have each test loop enough to get a good estimate of the probability distribution.