Overwatch Randomized Trial on Gender
danluu.com
danluu.com
Reminds me of a professor at Stanford who underwent a sex change and experienced very different interactions from his peers: https://www.nature.com/scitable/content/does-gender-matter-b...
We recently did a survey of 240 collegiate esports players and found that women were 9x more likely to be "very concerned" about sexism and 2x as likely to be "very concerned" about bullying in competitive gaming culture.
I think Dan's research can help explain why.
Unfortunately, there's a good reason why academics use proper experimental design. Without it, all you really have is suggestive noise, if even that.
Low standard has higher odds than no standard of evidence or no research at all. High > low > zero. By what logic could you claim that low = zero or zero > low?
But all else is not equal. The pressure to publish something interesting, even if false, greatly outweighs the incentive to be accurate, and overwhelms whatever filter the low standard might provide.
Even the author of this article acknowledges the problem:
> When I write a paper, I have to deal with co-authors who push for putting in false or misleading material that makes the paper look good and my ability to push back against this has been fairly limited.
To their credit, the author makes some effort to point out the flaws in this study, and acknowledges that this is just "a preliminary result that is, at best, weak evidence."
That's a start, but obviously not as good as actually putting in the work to correct those flaws.
CS:GO is a classic case of this - the player base is noticeably more mature at certain times of day, and there's a distinct period when all the kids have just got home from school where things are particularly bad. Occasionally you'll get a team where you everyone works together well, but to have that every time you'll need to spend a lot of effort maintaining a huge friends list in order to be able to put a team together at any given time - rather than just pressing the matchmaking button.
(I think you could might get decent results on this by having each player rate the others at the end of the match - but you don't use the ratings to rank the players, instead you'd use them to cluster players based on who they mutually like to play with. You'd then use that as a variable to optimize on when doing matchmaking. Hopefully with enough data you'd be able to predict if players would have a good experience even if they've never played together before.)
Dota has a system like this called "behavior score". I know it works based on reports, but it might include other things like text analysis or item feeding. It seems to work pretty well.
Not enough players. Any matchmaking system requires a large number of players to be playing at once to work. People are generally unwilling to wait several minutes or more to get into a game. The problem is that this has a cascading effect: the longer your queue times the more people will quit and the longer your queue gets. Below a certain threshold the system breaks down. If you start classifying people based on criteria such as social cohesion then you increase matchmaking time even further.
There are some games that could have a matchmaking system like this. But no developer knows in the development stage that their game is going to have this kind of popularity, which means that this type of matchmaking would have to be built after the fact. At that point you risk ruining something people like about the game.
Another thing to keep in mind is that player numbers fluctuate significantly. The number of players playing at 5 am between Monday and Tuesday is much lower than the number of players playing at 7 pm on a Friday evening. You would have to change matchmaking algorithms based on time of day or activity.
You would also have to change matchmaking algorithms based on skill level. At the very top end matchmaking is going to be long due to there simply not being enough players at that rating. This can have a negative impact on the retention of players, because popular streamers usually have a high rating. Watching a stream of matchmaking queue isn't very interesting and doesn't make people want to play the game.
Then there's the question of whether you even should do this. People make friends in games. If a game introduces matchmaking on social cohesion you essentially put players into a filter bubble. We see from Facebook and Reddit that these kinds of filter bubbles aren't ideal. I think one of the coolest things that I've got from games is that I've interacted with and made friends with people wildly different from me.
I've played hundreds of hours of Overwatch on console with my wife. We don't use mics, because we had enough bad experiences early in our career to make it not worth the risk. I never have problems when I play by myself.
Surprising number of people questioning the methodology in this study. I would've expected that if it ran counter to anecdotal experiences described by women (and men who group with them) who play, but I'm not convinced more rigorous methodologies would significantly change the results. If someone does believe that, I'd seriously challenge their reasoning.
The LFG system also helps.
Also FWIW the voice you use in OW can cause drama. I've definitely had games where I get flak for my voice - when it was more androgynous often people would want to know my gender.
I've seen people get exploded on for their accents, sounding young, etc.
The result did run counter to anecdotal experiences described by women, as is mentioned in the second section. The female account did not see more sexual or sexualized comments, out of the whole 339 tests only one was definitely address to him and a second event that the author is uncertain about. As describe in depth by the author, outliners like those are of questionable value. That is very far from the anecdotal experiences usually described about competitive online team games.
The questioning/challenging of abilities was the other finding, which the author speculate is derived from when a player is underperforming and get frustrated they lash out at team members who is overperforming. This is where a gender difference is found and it is an excellent starting point for further studies. For example it would make for an interesting path to compare studies that look at game theory experiments where player punish generosity, and in cases punish generosity more than defection.
I would also like to see a more in depth study on exactly what underperforming players do when they get frustrated. What is the common outlets and how they differ depending on circumstances. The author did mention that they were more often the overperfoming player when he used the female account so its possible the methodology in this study actually did influenced the result, which the author says in the article.
I found this study interesting but it did not change my view about the issue. I have always though that the difference in online game experience comes from the method people use when trying to insult someone else, rather than the rate in which people do it. If the only clue someone has is the name then that is what people will use in order to guess what method create the best insult, which results in primarily a gender difference.
Maybe it’s because I grew up playing FPS, but for as far as I remember people in these games were extremely hostile to women.
I will say, though, dedicated servers which were far more common back then typically had admins that would moderate servers. On good servers toxicity was almost nonexistent. I miss those days a lot.
I like matchmaking of today a lot too, but some of my best memories came from the communities built on dedicated servers.
I also played Starsiege Tribes and Starcraft Brood War online, and... yeah, a bit more toxic over there. The "hardcore competitive" community just oozes toxicity for some reason as far as i remember.
But even "Yahoo Chess" (a very peaceful community) would have a difference between my Sister's chat and my chat when we were kids. People really do treat women differently online. My sister was slightly better than me in Chess (she probably won 60% of the time vs me), but otherwise we both were roughly on the same level. The only major difference would be our usernames.
Some examples in my experience are games like Counter-Strike, Dota 2, LoL, etc. A couple common features of these games are:
1. Matches tend to be drawn out over at least 30 minutes, if not longer, forcing you to be teamed with people you may not get along with for a longer period.
(Overwatch comes close to having this issue, but I still feel the shorter match times tend to limit extreme toxicity you can find in other games)
2. A single player actively working to harm the team, or simply not at the same skill level as the rest of the players, can easily sink the team's chances of success.
3. An individual matchmaking / visible ranking system which fosters toxic attitudes like "well, I'll sink your rating if I don't like you", since every member of the team is ostensibly trying to raise their rank.
#3 when combined with #1 and #2 can create situations where "trolls" can easily ruin a game that may take an hour to finish and may even prevent any chance of victory.
These factors tend to be key to the most toxic game communities I've been involved in.
Back in the day a lot of cheat accusations came from people who really were cheating themselves (games modified to reveal territory and units which should be hidden to you), today who knows why they do it.
20 years on and it's still going because of community modding and dedicated servers. I'll eat my hat if Overwatch is available in 20 years.
I was much younger, and didn't know how to "think competitively" back when I played Tribes (or Tribes 2. I'm looking at the Wiki pages and I'm having issues distinguishing my memories between the two games). I pretty much was the guy who logged into servers, played as if it were "Deathmatch" (even though the flag was important and stuff), and eventually left without telling anybody. If there was a n00b staring at a console or something irrelevant, that was probably me.
Its a little intimidating for me to think of going back to this game, where the community has been honing their skills for the last 20 years :-)
Still, I'll probably check it out. Thanks for the info.
That happened in Overwatch sometimes too, but I was surprised it wasn't more often. My suspicion is that it's the a result of the competitive mode putting focus on the gameplay and a larger than average female player base for an FPS.
With OW, I'll hear a female voice at least a third or fourth of my random games, as well as see female-gendered screennames who may not actively participate in chat. I haven't played many games between old-school CS and TF2, so I don't know if the overall demographic has changed. But I wouldn't be surprised if OW's diverse and welcoming cast and lore was a major factor in attracting female gamers. As much as I liked the older games, OW was the first game that made me realize how much diversity could itself be a feature -- not just in attracting a playerbase, but just more creative designs (though I understand it may be too cartoony for some, of course). If only Blizzard's writing could be as clever and less clichéd-ridden than Valve's...
This isn't statistically analyzed, but I think it's a fairly significant detail that's not emphasized more in the paper.
> Per above, I didn't want to code tone of messages to avoid bias, so this table only indicates the rate at which people told me I was playing incorrectly or asked that I switch to a different character. The qualitative difference in experience is understated by this table. For example, the one time someone asked me to switch characters in the masculine condition, the request was a one sentence, polite, request ("hey, we're dying too quickly, could we switch [from the standard one primary healer / one off healer setup] to double primary healer or switch our tank to [a tank that can block more damage]?"). When using the feminine name, a typical case would involve 1-4 people calling me human garbage for most of the game and consoling themselves with the idea that the entire reason our team is losing is that I won't change characters.
Similarly, the opening preamble points out that often the worst harassment is potentially related to when players hear female voices. Obviously there is a different way you could do this study, which would be to invite ~300 players of various skill levels to play a single game and record the results for analysis. That type of study would have it's own problems, but you could at least be more likely to control for some of the problems in this study.
Anecdotally I've found a noticeable improvement since Endorsements, which this data missed.
Also the rise of Fortnite and more recently Apex Legends has led to a further decrease in toxicity that I've seen. A lot of toxic kids left for BR pastures.
That's on PC in the EU region on a high-silver current, low-gold peak SR account, casual QP-heavy play but some comp, pure flexing (as I prefer to fill rather than focus on maining one or two characters. Maining is arguably the best way to climb out to Plat).
I'm an almost ridiculously positive player with a gender neutral name and I've found being overtly positive vastly improves the experience.
It does prompt a couple of toxic responses ("kys", "fuck off" etc) but the majority is really positive. (I generally just report any toxicity.)
Playing without voice in Comp is sub-optimal, as winning is much easier with some kind of synergy. Even a basic PharMercy combo, or ultimate combos like a EMP+DVa Bomb can be highly effective.
I normally Solo Q, but playing paired with a friend in QP yesterday I had a surreal experience. One player on the enemy team insulted my friend (sexual but not specifically-gendered insult). She was playing with a non-gendered name, as Moira, and the toxic enemy player complained she was carrying us.
All the players on both teams turned on the toxic player, calling them out and highlighting why they were actually losing. I have not seen anything like it before but it was hilarious.
Stay As A Team is great in theory, but a team of 6 randomised will get rekt by a pre-made 6 stack on Discord with each other, and the matchmaker doesn't seem to take that into account.
I wonder what region author was playing on?
You really need to consider how to account for experimental variables that affect each other.
Important: this coming from my background, which is WoW, and not any of these modern games and communities. Maybe things have changed? The average WoW player is definitely older than the average Fortnite/OW/whatever player.
Certainly women play video games, and one only needs to talk to them and ask them why they choose one name over another to quickly accumulate first hand stories. If you do this with people you know in real life, you may find some clues as to what you aren't seeing first hand.
Edit: and, for what it's worth, you may forget that WoW has some unique rules on names compared to other games. No idea how that may affect things.
No idea how that may affect things.
While the GOTIS principle is something that many female players actively apply (they try not to signal that they're female to avoid attention), it's a tragic thing to operate on or defend.
The only reason it's salient is because most users of a community are male, which is self-reinforcing. Female outliers get naturally highlighted in a highly male context, and, whether or not they use this to their advantage, they'll suffer its downsides. It's tragic that they should have to and that we're still so sexually partitioned, culturally.
As a side note, I recommend meditating on why seeing a female name on the internet upsets you, when you should be well aware that 50% of human beings are female. If players didn't get upset and force females to play under male/neutral names, maybe it wouldn't be as shocking to notice when they don't.
Is this true broadly? I'd be curious to know if there's any data on that besides for anecdotal evidence.
also, a "neutral-looking" name can be really subjective, I would guess that there's a lot of player names that use adjectives that might be inadvertently related to a particular gender based on the cultural context.
I'd be curious to know what the names are in this case.
"Dragging your personal identity in" only seems to apply when that identity is different than the assumed default (male), hmmm....
Apex Legends conversely has been amazing in comparison. Maybe its because its new, the tagging system, that its a free game, or that there is no ranked structure (yet), but I've met nothing but cheery people and the occasional crude drunk. I've played with a couple shy women, but (anecdotally) have seen not one iota of what i've seen in Overwatch (aside from crude usernames that are prevalent everywhere).
Wait... Shouldn't it be the opposite? Paying for the game should make you more reticent to being an ass because losing access to it means you've lost money.
I wonder if this suggests that hardcore/invested gamers tend to be less kind than casual gamers or if this is anecdata.
I can't prove this scientifically, but yes absolutely of course "hard core" gamers are more likely than casuals to be raging toxic asshats in any kind of competitive environment.
I have even seen people admit to doing this. That's why you can't appeal to reason by explaining that their behavior is contributing to the loss. Once someone begins to rage, they have already decided that justice is more important than victory.
Dota has gotten better as a game as years gone by.
Not from my experience. I quit Dota around 2016 because the community was becoming too toxic. I think MMR ranking system they introduced was a big factor. For me the sweet spot was 2013 to 2014, the game felt a lot more fun and casual.
I was never much good at the game but I used to play a lot 3 hours+ a night every night for almost 4 years. I calibrated at 2k MMR when they introduced the system (which is very low) and over two years slowly grinded my rating to about 3.8k at peak. I found that higher I climbed worse the attitudes of people I was playing with became.
It got to the point where I was deliberately avoiding solo queue and only queuing in party with friends until I started logging in less and less and just stopped all together - one day I was sitting at my PC about to log in and just kind of had this epiphany 'Dota isn't fun anymore, why am I logging into this game where I know people are going to insult and belittle me.' I haven't played since. I still have a lot of good memories of early years of Dota but I kind of look at it with horror about what that community morphed into.
I disagree with the point that dota 2 had a nicer community. The game grew out of Dota 1 and many of the attitudes from that game carried over. It was standard practice for new players to be called noobs. Then there's the infamous "Welcome to dota. You suck."
Try solo queue comp CS:GO. CYKA BLYAT, IDI NAHUI PIZDEC.
I wasn't a particularly good player, but as a 1.6 enthusiast, and a CS:S addict (on a brutally admin'd clan server), the rampant cheating in CS:GO was game-ruining for me.
I don't mind eating lead from a skilled player, but cheating just sucks the fun out.
Not to say OW is without problems, obviously. Not related to OP is throwing and smurfing. I was in a Stay As Team group that got dumped into a large group of smurfs. They let us cap and win and didn't even contest, but would occasionally just mess with us to show off their mechanical skill. Most of them could have 1v6'd us. Most of our team reported them, but it was just really disappointing.
OW definitely has problems, I just find them less than other games.
What region were you playing in, and have you tried others? I get that ping might be an issue, speed of light being uncooperative and all, but I'd be curious to know your experience based on region and what times you play.
I think it's North America.
I also think it will vary game by game. I don't play a lot of games. Most are shooters like Modern Warfare, Battlefield, and Mass Effect 3. Modern Warfare 2 and 3 on Xbox brought in more trolls in voice chat than all my others combined except maybe GTA V. In the last one, you play as a vicious thug causing problems for everyone around you. Them being pricks makes more sense. The difference in shooter games, though, was more interesting given it might reflect different sub-cultures in gaming. Specifically, MW is designed to appeal to largest number of people with lowest barrier to entry and patience required. Might self-select for higher percentage of jerks. Whereas, Battlefield 4 and Mass Effect 3 push people to work together more for best results.
Heck, even a "casual" MMO like PvE Guild Wars 2 games, have chat-rooms and voice-chats because its way easier to specifically say "My Rocket Boots are charged, can anyone drop a light circle?". And that's not even truly competitive or "hard" compared to FPS games.
Go into PvP Guild Wars 2, and you won't get anywhere without voice chat. Coordination is a huge element of the game.
Maybe the other players were just really good and I was getting carried? But from my anecdotal experience, good voice chat tends to add a ton to team momentum.
With someone who's playing TV in the background, someone who doesn't bother with callouts, and someone who's tilting hard... I'd rather mute them and myself.
I think it's notable that although these are not directed they are highly gendered. This is part of the problem as well.
It would be interesting to see what roles the author was playing, and the exact SR of the accounts, as there seems to be a great deal of variation in what people will say on voice comms (usually higher SR, because voice comms > chat) and what will be said in chat. People also like to blame DPS first, then demand a Reinhardt (if they don't have one) and then finally blame support.
Without hard data as to what is going wrong with a team composition, toxicity is almost guaranteed when losing.
As someone who mains support, I don't think this is true. I've had plenty of games where I do a shitton of healing, get the healing card at the end, and still had people blaming me for "0 heals" all game.
A scoreboard would help with that, but it'd be a trade-off with having 5 angry people shitting on you if you ever have a bad game. I'm not sure that's better.
EDIT: Plus there's all the times when your stats are bad but you're not the right one to blame, like when the whole team can't move up through a choke because the tanks are too chicken to go first, but it gets reflected in poor DPS eliminations.
Ultimately it's a team game, and failures of teamwork aren't reflected well in a stats scoreboard. I think you'd incentivize poor team play by people trying to pad their own numbers at the team's expense.
And no matter what information is available, some people are just looking for someone to blame when they lose. I'm not sure how to fix that. If matchmaking is working right you're going to lose about half the time, there's no point in getting worked up about it but people still do.
Scoreboards can also give people license to be toxic. If you're on the losing team in CS:GO, it's pretty common for the top fragger to start cussing out the rest of the team as noobs, even if that player is greedily hunting frags and the other players are making a useful tactical contribution.
Because if not, the whole study seems sort of moot.
I wonder if there is software that can convincingly change the gender of a voice in real time? Could make for an interesting experiment.
The above experiment with neutral names would give some insight if "gamers" do indeed "default to male", as you express it, or not.
> I avoided ever using voice chat and would call things out in text chat when time permitted.
Again, this betrays a deep flaw in methodology, since voice chat is an integral part of competitive Overwatch. The important things to consider here are: what percentage of players are comfortable using voice chat and how does that vary by gender; does availability and usage of voice chat confer any competitive advantage to the player; and do the two factors interact in such a way as to confer a competitive advantage to one gender over the other?
I find it disturbing when men do scientific studies to determine if women's stated experiences are real.
What, in terms of profit? (And in the next paragraph, competitive advantage, really?) That’s a very narrow view of what it means for something to “matter”.
This post is useful because it is a well-documented experiment that proves the difference between male-usernames and female-usernames online.
women as a group do not have "stated experiences". individual women do, and if you talk to enough of them you'll find out that they have very different experiences and opinions.
most women I have talked to who play competitive video games do say that they experience gendered harrassment, and I don't see any reason to disbelieve their individual accounts. on the other hand, I don't actually know very many women who play these games, so my sample is pretty small and possibly skewed by all the filters that influence whom I am likely to have a conversation with.
if I cared a lot about this specific issue and wanted to collect more data and apply stastical techniques, why would that be disturbing?