Comment by triska | Hacker News Reader

Comment by triska | Hacker News Reader

Parent Full thread

Complementing this article, there's also a great post mortem article about Age of Empires:

https://www.gamedeveloper.com/pc/the-game-developer-archives...

In the section "Things That Went Wrong Or We Could Have Done Better", it mentions as the final point an insight I often think about:

"8. We didn’t take enough advantage of automated testing. In the final weeks of development, we set up the game to automatically play up to eight computers against each other. Additionally, a second computer containing the development platform and debugger could monitor each computer that took part. These games, while randomly generated, were logged so that if anything happened, we could reproduce the exact game over and over until we isolated the problem. The games themselves were allowed to run at an accelerated speed and were left running overnight. This was a great success and helped us in isolating very hard to reproduce problems. Our failure was in not doing this earlier in development; it could have saved us a great deal of time and effort. All of our future production plans now include automated testing from Day One."

Very clever but it only tests actions the AI takes.

It is true that actual interaction is not tested in this way. Still, AI actions are not all it tests: For example, just by keeping such games running for a long time, it implicitly also tests resource management (such as garbage collection), robustness of the networking code, synchronization etc.

What I found appealing in the description is the idea of keeping your program running for a long time and have it perform all kinds of actions automatically. Nowadays, I often hear about "test-driven development", "unit tests" etc., and it often turns out that they test very specific things out of a vast universe of all possible things. That does not mean that they are useless. As I see it, it means - as they phrased it in this article - that we often do not take enough advantage of automated testing.

Personally, when I test software, I always try to follow the general idea stated in the post mortem: Keep it running, and perform all kinds of actions automatically. I found several crashes and memory leaks in this way, which were not noticed during manual interaction because they only became significant when the actions were repeated thousands of times.

The AI actions are the same actions that players would take manually, they just get computed by the AI instead of a brain.

Afaik the AI logic gets processed on the host instance, and the rest just follow the commands sent through the network.

Based on my personal experience of automated testing in AAA games, even if you "only" count the overlap with the actions humans take, it's a huge overlap in practice. And while there are edge cases only humans find, automated testing finds bugs humans don't.

The point of that section is that the earlier you find bugs, the quicker you fix them, and you end up with less overall pain for the length of the project.

Reply on news.ycombinator.com