Hacker News Reader

Comment by kyberias | Hacker News Reader

Are these all the automated test they run? The article doesn't seem to mention unit tests for example. Do they write and run unit tests? The test class example seems to assert a bunch of stuff that might be easier (and cleaner) to test in a set of unit tests.

Update: In particular this is weird:

> "Tests make use of remote procedure call (RPC) endpoints exposed on the client and the server in order to issue commands and monitor game state. For the most part, tests consist of a fairly linear set of instructions and queries—existing tests cover everything from champion abilities to vision rules to the expected rewards for a minion kill. "

They test "rules for expected rewards" with an out-of-process python test program that connects to the client and server via RPC. Seems unnecessary complex way to test specific game rules.

Is this a symptom of a separate QA team and no developer-written unit-tests?

View on HN

No, it's because game engines don't unit test well. There is too much I/O, latency dependent behavior, and shared mutable state to make it feasible outside of some math and low level format or protocol tests. Many game bugs are purely data bugs and result from a misconfiguration or a malformed asset. Manual testing of the result, following a checklist and creatively hammering on the system, is thus the default method.

What Riot is using is a form of integration test where manual user input is emulated to produce a result data set. What you aren't seeing in the test code is "everything else" that was needed to set up a running game state. This technique makes it easy for QA to jump in and see what's happening visually when a failure occurs, eliminating the need to slave away at a checklist.

It's useful as a framework for testing all game interactions. When you think about the amount of unique abilities for the hundred(?) unique characters and how they interact against eachother, the test matrix that results is massive.

Additionally, you want to use production server/client code as much has possible, while getting through the tests as quickly as possible (skip front end flow, matchmaking, etc.).

Using client/server endpoint calls are great because it allows these integration tests to be a list of instructions (the same instructions used by production code) to create a very deep test suite.

Integration testing is critical for games, where you have so many independent units interacting and changing each other's states.

>> When you think about the amount of unique abilities for the hundred(?) unique characters and how they interact against eachother, the test matrix that results is massive.

Definitely. They're at 130 champions right now. Each with four spell cast abilities, as well as a passive ability, and unique auto attack mechanics on some of them. There are indeed many interactions with specific abilities between champions. Add in interaction with the map and terrain itself, like unit collision and NPC minions and monsters. Then deal with the 154 items (the current count on the main map) players can add to their champion, many of which also interact with abilities, and some of which introduce additional abilities.

It must be both a nightmare and yet very interesting to handle creating a new champion for the game. Once they get past the step of even deciding how the new champion's abilities should interact with other champions, they then need to code it all and manage to verify that everything works according to expectations. Every once in awhile, there are still bugs with how one ability interacts with another. You can plan it all out, but it has got to be easy to miss something. Too many interactions! :)

That's why the BVS they've built becomes so important. The amount of test-cases you have to satisfy go beyond the human mind's ability to hold.

But if you have a farm that tests v1 of your new champs abilities against all others, you know exactly where the 20% of problems will lie. This is where designers earn their paycheck- two abilities that won't resolve by design, and where the programmer puts 80% of their work.

80/20 rule scales with the product, I suppose :)

It's a more loosely coupled way to test than unit tests. That has certain advantages.

Reply on news.ycombinator.com