Hacker News Reader

I think games are hard (if not impossible) to test with unit tests or codified tests like the ones you mentioned.

At the company I work for we rely mostly on manual testing for testing the games and applications we make. We do some automated testing in the form of Tinytask tasks that are left on overnight to hammer an application. In terms of release testing, especially with a custom game engine, there's no easy way to codify a test which actually plays through a game where there is some random mechanic, or checks for things like the visual accuracy, windows flashing up on the screen when they're not supposed to, etc.

Web development frameworks like Selenium are great for UI testing, but they require identifying interface elements by their IDs, or performing a 100% image recognition match for targeting. If anyone knows of a good UI testing framework that would work for games/DirectX apps, I'd definitely love to hear about it.

I'm not bagging automated testing, I think it's a great way to save time and avoid the boring bits. I just don't think it's very relevant to game design, but for the OP's enterprise app job it would definitely be suitable.

I think the #1 reason why games don't need a lot of automated testing is serialization, or rather the common lack of it in most game worlds. If the data gets corrupt, you can reset the game and all is well again. When the rare corner cases pop up, it's easy to play whack-a-mole with them, because they're all "shallow" in the sense of the game starting from a clean reset frequently, so you can usually reproduce the bug quickly.

Of course, if the game does need heavy-duty serialization of everything, all bugs are potentially deadly. And this is largely the case for the line-of-business, social, and productivity apps, because that data is considered mission-critical. Corruption is not OK, reverting to backups is to be avoided. When a game has that kind of requirement, things get a lot tougher - and so games have naturally evolved to favor minimizing the save data to nothing or a few stats.

And this is borne out by looking for contrapositives. There are two genres that have a history of tending to be buggy because of some form of long-term data corruption: Large-scope RPGs and turn-based strategy games. It's hard to reach the bugs in those games, so it's also hard to fix them.

This is the difficulty for the company I work for. We make a range of desktop applications including launchers, productivity apps, media apps, games, etc.

Because all of these apps are based on a game engine, we get the complex and hard to test bugs that games get (texture cache corruption, null pointers in the scene tree, etc.). There's much more serialization and pressure than with pure games (but still not as much as with enterprise applications) because we're dealing with transactions with 3rd party services, or because a screw up could corrupt a user's whole music collection, or because users don't expect their music player to crash every 4 hours.

The game engine adds a huge amount of extra variability to our applications; we have to not only watch out for obscure bugs in our custom script code (which the engine silently tries/fails to execute anyway), but also in the (C/C++ based) game engine it is driving. The upside to using an engine with a simple scripting language is the shorter development times to get things off the ground and the high-performance/shiny visuals, but I feel it costs us much more down the line in terms of stability and extensibility.

Wait, wait... "texture cache corruption", and "null pointers in the scene tree"?

Edit: It's both scary and awesome that you rigged a game engine to manage a music library a-la iTunes. Are you profitable?

I'm not sure I would be allowed/qualified to say much about our financials. From what I can gather we are profitable, with ongoing contracts for clients such as Dell.

http://www.unlimitedrealities.com/blog/video-the-dell-stage-...

Those issues I mentioned are examples of things that we have actually run into.

edit: Our specialization in touchscreen development is really what drives the company, but hardware accelerated graphics are also a huge draw.

At the risk of going too far off topic and without intending to be too harsh: at least in the video, that interface looks nice, but there is too much lag between when a tap/gesture is made and when the interface reflects it. This breaks the direct manipulation metaphor and is, I think, a showstopper. The pinch to zoom example was particularly off-putting: how can I know how much I'm zooming when the interface doesn't track my gesture? I cannot imagine being happy with pinching, waiting a second to see what's happened, then pinching and waiting to adjust (repeat until I get it right or am too frustrated).

I don't think there's too much danger in going off-topic if the conversation is interesting and has some substance.

Yea I think lag in touchscreen interaction is a huge problem. I think this is largely a result of hardware. We've had to ship on atom-based hardware with 2003-era DX9 graphics. Even the latest Intel chips with DX10 are horridly slow when it comes to graphics.

The other side is actual touchscreen hardware. The machine used in the video is an HP Touchsmart, which uses an optical touchscreen panel. These panels have a latency of around 100-200ms, and that's before we even start processing the touch event information. Capacitive touch sensors are much better, but the're expensive to manufacture above about 10 inches.

In the end lag/accuracy is a reality of the low-cost hardware OEMs use, and there will always be some trade-off between hardware cost and performance.

I am going to have to disagree with you on the unit test part. Everything in a game that is not graphics has no inherent limitation that prevents automated unit testing. A large part of your graphics pipe line can be unit tested as well, most of the scene graph operations, for instance, should be unit testable. Sure you could still end up with display bugs do to driver interaction, but you should still try to lock down what you can.

I think you can use automated testing for games, usually... It requires a few compromises.

If you're feeding input into a non-deterministic game the character won't be exactly where they were last time, etc. You could keep track of the movement offset and see if it's within an acceptable range, or you could check the character's speed at two times and make sure they're accelerating properly, etc. Design a test level to highlight the potential problems (weird ground polys, whatever) and write an in-game script to test the behavior.

I think I could even script visual checks - mostly. You'd make, for example, a level prone to Z-buffering errors, where the colors were chosen to make it obvious - like a red wall showing through a white one. Save a stream of screenshots and compare them. Trivially, check for red. More complex, and perhaps better, check for high-contrast areas. It wouldn't tell you if the picture looked good overall but it'd be fairly good at finding instances of that one bug.

And for unit tests, don't be so quick to write such a good test that throwing it away is painful when you want to refactor the code.

Given that the op mentioned that he was an Android developer, it's worth noting that the Android SDK possibly has the very worst testing environment of any widely used system on the planet ever* .

* possibly an exaggeration, but it's definitely very very bad.

So I was working on developing a multicast serial driver. Since I was developing the serial driver, I had to take... over... communication... from the copied-from-ROM driver.

And since there was only 1 working communication port, how did I debug? How did I even run a test?

The board I was developing on was amazing. It had 1 (one) LED.

Blink.

Blink (fast).

You should have just taped a photoresistor over the LED and serialized your error codes back into your dev box with morse code.

But I guess then you'd have to debug that code and you'd be back where you started.

This setup is almost exactly like the one in which I first learned C (it had a massive 2048 bytes of SRAM!).

As a bonus, it used a dodgy proprietary compiler with a whole sack of undocumented "technical limitations."

The good part is I now have a really good understanding of pointer manipulation. The bad part is using it sometimes makes me twitch.

Edit: Here's someone who's got us both beat: http://web.archive.org/web/20070613032334/http://ipodlinux.o...

http://3.bp.blogspot.com/_AxcE8NTHeDU/TOW6kRd-FGI/AAAAAAAAAy...

What specifically would you like to see in terms of testing tools for Android? I ask this not because I don't think there are areas to improve (there are), but to get a better idea of where improvement is most needed.

Java has good unit testing frameworks. Try Obj-C/Cocoa.

you're talking about Test-Driven-Development (TDD). Any good resources for integrating TDD into javascript based games? I'm still looking for that gem.

To expand on testing a bit: at one end, there's unit testing where one tests maybe a single object's behavior in the game. I think for game dev, testing for absolute values are not as important as deterministic ones. We would rather want to know if a player has jumped, died or shot something.

Eventually, a functional testing is needed when objects interact with one another. It's like a system of systems. That one's more complicated to design. It might go as far as designing a test that runs through a complete level within miliseconds. Anyone have ideas/resources on this?

I am well aware of testing, TDD, BDD, and have been plugged into the scene for a while.

For the server and client both, I really like Jasmine as a testing framework. Of course, there's always cucumber :)

http://pivotal.github.com/jasmine/

> This forces you to ensure you have good test coverage (every feature is tested)

And it tests the tests too!