Hacker News Reader

Comment by jayd16 | Hacker News Reader

I don't want to weigh in on the rest of the content but the characterization of the game industry is pretty accurate in my experience.

I would expand more on the first bullet point of why game devs don't test. Tests are anti-agile and game development is extremely agile. Usually you don't know what kind of game you're making until you're done.

View on HN

That's not true. From my experience unit tests are great for agile. It will allow you to create "trusted" modules which you can move around and rework much easier (you can also treat your tests as executable documentation). Without tests you can't safely do any change to existing code especially if you are changing code written by someone else. You have to risk it and than spent considerable time in debugger if it breaks something unrelated to your feature.

>That's not true. From my experience unit tests are great for agile.

Not when you're constantly prototyping, which is what game dev essentially is for the most part of the process...

> which is what game dev essentially is for the most part of the process...

Except that it isn't. You don't maintain the equal velocity of changes throughout the process, even for indie games. And there are always portions of the game amenable to tests.

Changes until the last minute to crucial gameplay elements are not uncommon..

Its not like tests are impossible. It's more that TDD is almost impossible.

Sure, menus and networking. Maybe loot distribution, level generation, etc.

Not so much actual gameplay though. That does constantly get tweaked.

I'm not sure what most teams do now-a-days but I went to GDC in Koln and saw the Croteam talk. Over 10 or so years by programmers just adding a little here and there as it occurred to them they had build a pretty cool testing system.

First they had made it so if someone was playing the game and saw a bug they could press the "file a bug" key, type in a description and the game would save out enough info to bring someone back to that point in the game, same camera, and possibly other state. From the bug database they could click a link that would launch the game back into that state, let someone verify the fix and mark it as fixes.

The also had a waypoint system for bots to play through the puzzles (this was the Talos Principle they were talking about). If the bots ever got stuck, as in didn't make to the next waypoint within some time limit they bots would file a bug using the system above.

https://www.gdcvault.com/play/1022784/Fast-Iteration-Tools-i...

As another interesting idea apparently the creator of Thumber built a URL system so people press a button which would generate a URL into the clipboard, they could then paste that URL into Slack (or email/chat/etc) that would launch the game in a particular state to pass that other users on the team.

Similarly, on a game I worked on we had a “controller monkey” that would spam all possible player actions (move, jump, attack, special powers) rapidly and randomly. We then had the testers record multiple paths exploring through each level. Every few seconds the monkey-controlled player character would be teleported a bit further down the path to ensure progress and coverage. Dozens of people would set the monkey to run all night when they went home for the day. In the morning we would have a dozen fresh crash dumps.

I feel like when people just say "tests" there's a lot of conflating. Usually, on a site like this or on a blog post, "tests" are referring to automated unit-testing or talking about TDD...which is probably most rare and most fought against in game dev (from what I've seen from an adjacent industry).

Testing, in general, is pretty essential to writing code that does what you want. Test code is just automating what you'd be doing manually and it's a lot faster to have the code do it than for me to do it 10 times. Even if that's printing out a value or showing it in a debugger. Testing frameworks or libraries to fuzz out problems and harden code are quite common. Game dev often has a lot of manual play testers. Most engines and dev consoles have a lot of tools to either just record the screen or save state when problems are seen.

Heck, most "cheat codes" were added to jump around the came to test for bugs. That's technically "test code."

> Testing, in general, is pretty essential to writing code that does what you want.

Shipped 17 games, several AAA games with no automated testing. Not saying that was good only that it happened and so is at least some evidence that automated testing is not essential.

The problem with automated testing in games is most of it will be content specific and the content changes multiple times a day. A product like say GTA5 has 20-30 programmers and 300 artists and game designers. Those 300 artists and game designers are adding new content and or changing old constantly. Often they also use a scripting language or a blueprint like visual language to setup in-game logic like "open blue door if player has blue key". There's just too much to test. Every day 1000s of tests would have to be re-written because some boxes, doors, etc were moved .5 units to the left.

There isn't zero value in automated testing (see above) but my experience with testing on a large project, Chrome, was that it really slowed down my velocity. Easily by much more than 50%. Of course Chrome is a platform and absolutely needs the tests IMO. I think a game engine would benefit from lots of automated tests. The game itself though it gets harder to figure out where the balance is between automated testing and manual testing.

As others pointed out as well big teams, like an OS team or a browser team often have dedicated staff to setup and maintain a testing infrastructure. Game teams rarely have this. Maybe they should but few games are given the budget.

I agree /w testing game development being less of a priority than in other industries, but I feel that it is because code quality tends to take a back seat in gamedev.

Tests are NOT anti-agile, that is just dumb. I feel like a bunch of hacker news hipsters read an article about TDD 3 years ago and then said "Yep that's my opinion! Tests are bad." Despite every major software company requiring unit tests for their production code-bases. ( Hint: It's because they did the research and found tests beneficial. ) Tests enable agility.

Let's just get this out of the way now: Tests are not about catching bugs. Tests are about allowing your to safely refactor your code without breaking previously declared behavior.

Testing enables you to iterate and refactor code without constantly releasing new regressions. Testing IS code quality. If you lack tests you lack a core piece of code quality.

A sufficiently big game in an established genre with in-house engine and expansions has several levels of automatic tests.

In my experience, such games only have manual testing, done by the people who bought the game.

My experience is from working on one of the games with automatic tests.

From the playing experience: yes, there are way more games with internal manual and external community testing.

Yeah at a certain level game productions will usually have automatic “smoke” tests for general build stability and I’ve worked on one that had automatic feature tests with replayed input. These were generally useful for catching obvious crashes and regressions, but the overhead only makes sense for a certain level of production. Could also see, and have heard of, more rigorous functional testing of things like a procedural generation pipeline that are otherwise harder to get sufficient coverage of manually.

I think the reason is that tests have a very obvious up-front cost, while the time they save is distributed in the future, in a non-immediately obvious way. I still think that they end up saving time, with some exceptions like UI code, which are more easily tested "by hand".

Game project managers are infamous for not being great planners, so it wouldn't surprise me that they dismissed automated tests as "a waste of time" or "something that we can't do now because we don't have time now" (so we end up wasting more time in the end, having to do death marches, etc)

Modern games have only increased the benefits a game studio can gain from testing as well. Games are now moving into service territory which only increases the amount of time spent maintaining the game while continuing to add to it.

This. Determining when the effort should be applied is the tricky part. Games are still hit driven and get cancelled/re-purposed during development. You can spend a lot of QA engineering time developing systems to test functionality that never ships (case in point would be Fortnight - the original shipped game did not need to be tested against the current 100 player game instances and huge load but they could have spent a bunch of time testing AI systems that are no longer any part of the game).

I don't disagree that it's important to understand when something is purely a proof of concept vs something that will stick around to evaluate the costs. However, the AI systems are still in Fortnite (and they even used those systems for the Haloween event). The major money making part of Fortnite has been the battle royale mode though. If they open up the main game to be free-to-play similar to the battle royale mode those systems will probably end up being used quite a bit.

Or perhaps game projects are extremely difficult to manage? You think if it was just a matter of competence then these companies would put billion dollar revenue on the line not hiring the best they can find?

In defense of game project managers (a phrase I may have never said before) is that their planning is often fine, but hampered by changes to game design/direction/requirements. Unless you're churning out a copy-cat derivative game design requirements have to be reasonably loose as you find what does and doesn't 'work'. Finding the fun is not easily managed.

I mostly. Agree. Tests can be useful in some specific games, specific cases, but in general they're much harder to do in gamedev compared to other areas. Maybe one case for gamedev would be to test the calculations of character damage in a RPG based on several factors. But that is an isolated case.

Reply on news.ycombinator.com