Hacker News Reader

Comment by soltanov | Hacker News Reader

[flagged]

My Bevy game (released on Steam) has ~37k lines of Rust, and is stuck on Bevy 0.15 for this reason. It's not been a big problem for me though, because Iv'e ran into very few Bevy bugs (none that forced an update) and I was deep enough into production that I had already solved all problems with the 0.15 feature set. Overall, it's been a pleasure, and at the time of the 0.14->0.15 upgrade my codebase was small enough that it was done in an hour or so.

You could probably point claude at these release notes and have it need no further instructions.

Most of the time, the time you spend on these upgrades isn't "I spent 10 hours rewriting code to use new APIs" but rather "I spent 1 hour rewriting the code, 10 hours testing", and no LLMs can do those 10 hours of testing properly yet.

Sure they can. Make the AI play the game pre and post bevy update. And difference down to a pixel can be flagged. In fact you can get hundreds of hours of testing this way using sub-agents.

Sure, the AI plays the game, says everything makes sense, does this mean it's coherent and consistent? No.

I'd urge you to try this yourself and you'll see how much they miss and misunderstand. They're nowhere near to being close to be able to do that sort of playtesting themselves.

Any game that been 100% made by LLMs in a vibe-coding way where the author does no testing themselves, will guaranteed be an absolutely mess and incoherent. This is the state of the art today at least, who knows what tomorrow will bring.

You’ve taken my statement, that an LLM can help test a game by running hundreds of hours of gameplay equivalence between versions(which btw means the AI doesn't say "it makes sense", your equivalence checker does), and stretched it into a claim that I’m saying games can be made entirely by LLMs through vibe coding with zero human testing. That’s such a blatant misrepresentation of my argument that I struggle to take your comment in good faith.

The difference would likely not be down to a pixel, either by dependence on how a model provider feels that day, the hallucinations of the day, or just some odd tangent that would get solved by ~100x tokens for verification. The hundreds of hours of testing could all effectively be 1===1, but it all looks green.

You have the model play both versions of the game in the same way at the same time. The equality would of course not come from the model, it comes from your testing harness. You just use the model to automatically play the game. SO you are checking game version A(inputs) == game version B(inputs). Where the AI generates the inputs that play through the game 10 times in different ways for example.

The testing harness would have to be updated per game feature and to accommodate model tendencies or quirks. This sounds like regression tests with llms and some extra steps.

Video games are pure functions of list[inputs] -> list[output]. Where the input are timestamped HID events and the outputs are video and audio, and in some cases stuff like rumble. The equivalence suite can treat the entire game implementation like a black box. Let AI generate millions of sequences of inputs that play through the game in various ways. Check for equality on the outputs with the same game but on the prior bevy version. Simple, very effective. Slight quirks game be dealt with as needed by allowing slight timing mismatches, slight or specifically allowed pixel differences allowed under certain traces etc.

I'm really not sure what was not clear about my statement of "Make the AI play the game pre and post bevy update. And difference down to a pixel can be flagged."

The breaking Rendering changes have always been the big challenge, especially with external plugins that need time to migrate too. They're starting to get less disruptive over this past year.

It's been smooth sailing upgrading for me. I think .17 was the one with the most pain for me? It's been a while but things have been relatively stable.

Reply on news.ycombinator.com