Scaling Guild Wars for massive concurrency
codeofhonor.com
codeofhonor.com
Simulating players gives you the benefit of end-to-end testing much more easily (for example, it tests your login flow and disconnect flow, both of which are often written without much care for performance). It also practically demands that you write a scripting console for your game, which is an invaluable resource for error testing, debugging and development both by your engine and content developers. It's true that human players are more unpredictable than bots, but that's not a drawback, because you want to stress-test the hottest part of your game. For these purposes, you can write a small test level (think an arena) and have bots spawn at either end and run towards each other, attacking until they die. This isn't at all representative of how players will play the game but it will very definitely catch performance regressions.
Replays depend on two factors, deterministic playback and unchanging content. Deterministic playback is possible (though extremely difficult) to achieve, but it really has to be a priority from the beginning, and it has to have its own testing suite and fixing regressions has to be a priority. It's impossible to retro-fit it onto an existing game (sometimes by design, as non-determinism occasionally is mistaken for randomness). Unchanging content is, obviously, unlikely to happen during development and is much more likely to be possible during betas, but during those time periods you have much more time to devote programmer and content developer time and attention to ancillary features like expanding bot programming.
My understanding is that the replays were possible because the game engine was architected from the beginning with the server 100% in control of game events and the clients relying on message-passing to communicate with the server. This means that the recording captured on the server is a definitive recording of what actually happened in the game. Clients could desync, so a replay might not reproduce that, but otherwise they were pretty accurate.
I'm not sure how the unchanging content problem was solved. I think everything was revision-controlled aggressively enough that it would have been possible to set up a local server instance using old content, and it's also possible that most ordinary changes (altering a texture, a map, etc) wouldn't actually change the results of a replay. Definitely a challenge, though.
Replays came in handy later on for other features (like letting players watch others' PvP matches in-game) so they probably ended up carrying their weight in terms of development effort. I do wonder if they felt like a burden initially, though, before their value became clear...
I disagree with him on bots, though. CCP has a highly evolved bot system which is tied into their build system, so that unit tests can include player behavior. This isn't directly relevant to load testing (although CCP does that with bots as well), but it certainly demonstrates why bots are worth the effort.
And if you've got a full-fledged QA department, bot writing is a good QA engineer task. This is also a decent way to give ambitious QA guys a coding task that doesn't have the potential to affect customer-facing game play.
Without violating NDAs I can say I've worked at MMO companies that found bots invaluable for load testing. Replays will definitely catch things that bots won't, but if you're thinking about how many players you can fit in a zone, etc., etc. bots are great.
Game recording is established as a favorable way to make games better, to avoid a crappy beta experience or problems right after launch.
How are you going to get decent recordings in the first place? Playing with a couple developers seems a lousy way to jump start a good selection of games, is it? How is this chicken and egg problem solved? I miss a way to bootstrap the process, load a good set of (varied, representative) replays?
In particular QA recordings got used a lot to reproduce particularly tricky bugs and race conditions, since players usually weren't savvy enough to spot them.
The volunteer tester group was also useful for load testing, since it gave us a way to throw a hundred or so players at one of our development servers and record all of them simultaneously. It gave a reasonable approximation of what the load might look like on a single production server and it meant we could also capture recordings of how the servers (and clients) behaved under load instead of just having 4-8 developers playing on them.
[1] Done right, you can play a minutes long recording back in seconds, which is very useful for trying to fix a bug and verifying it against a known-bad replay.
The situation we're running into at my job is with double density servers (i.e. Dell C6105's). The problem we're having with heating double density servers. Maybe somebody with more experience than me can explain why cooling 1Us would ever be a legitimate problem?
...
Maybe somebody with more experience than me can explain why cooling 1Us would ever be a legitimate problem?"
The situation in server labs, hardware and infrastructure-wise was much worse seven years ago. Is it that hard to imagine?
Similar example: there's a tier 1 network provider with a solid datacenter business; I had servers in one of their facilities during the period in which blade servers were becoming common. They literally ran out of power; we had some empty racks with a contract allowing us to turn on power for them at any point and they couldn't fulfill it.
Looking back on the specs for 80s and 90s datacenters can be really amusing in retrospect.