Hacker News Reader

Comment by senko | Hacker News Reader

I love testing the new models by asking them to code a toy RTS game. Here's what Terra did: https://senko.net/vibecode-bench/2026/rts-gpt-5.6-terra.html (one try, in codex app, xhigh effort)

Comparing this to other models, I find it similar to GPT-5.5 and a bit behind Sonnet 5. You can see how other models fared here: https://senko.net/vibecode-bench/ (you can also fetch the prompt and the the 5.6 Terra resulting code on from that page).

I don't have access to Sol yet (on a Plus sub, which should get it according to what I've read), so can't do the more interesting test. I'll update the above page as soon as I get access - hopefully soon.

View on HN

So the measure of a model is how well they can recreate something they easily have thousands of examples of in their training data. There's probably a better base RTS on github somewhere for free.

>There's probably a better base RTS on github somewhere for free.

I... I think you are missing the point.

Well, it is a silly test, not a scientific benchmark.

However, I would say it is a measure (not the measure). If you look at the entries, there's a lot of variation - definitely not something they memorized outright.

And the test itself is deceptively simple. You need to do canvas rendering, there's pathfinding, command queueing, terrain generation, etc. There are some subtle click handler bugs (various LLMs often stumble on those). And I ask the model to do it all in one file, further increasing the complexity of the task.

And the result is something that you can instantly evaluate. And if the result is any good, even play! So yeah, I think it's a fair test.

I'm sure it'll get saturated at some point. Actually I started with Minesweeper and switched to RTS last December, because Minesweeper was being saturated. I'm expecting (hoping?) the RTS test will last until the end of this year...

The goalpost velocity is approaching light speed…

Ask it to generate a completely bananas game idea and it will easily do it. Prototyping any type of game with simple graphics has been solved since many generations of models back. Claiming it's because it has that game in its training data is nonsense.

The 5.6 model article for this post has three examples of little web games.

There are plenty of little js web games anyway. The point isn't to make an actual game, it's to show coding ability, design and taste in a way that's more assessable than reading a codebase.

this is so cool: it's playable (even though super boring since there are no enemies) and you can feel that a few iterations would make it very usable.

Which model is the best at the moment, for this kind of stuff, in your experience?

I'd say Fable 5: https://senko.net/vibecode-bench/2026/rts-fable-5.html

It even has enemies! (I'm not too mad about it not following my instructions because it can be fun to play :) And I generated that from Claude Code on my phone.

Sonnet 5 also produced a pretty nice version. You can see all of them here: https://senko.net/vibecode-bench/

> I'm not too mad about it not following my instructions

I'm a bit concerned about this - starting with GPT5, AI labs started doing this 'complete app from a prompt' sizzle demos. When I started working with GPT5 - which was supposed to be a qualitative jump, just like Fable is now, I tried to do a frontend, and discovered that it gave me a CSS-animated purple-blue interface with embossed buttons, gradient backgrounds and dropshadows.

It looked very cool, but it was a bit overwhelming (also broken), and I was really looking for a pedestrian Bootstrap job.

It required not inconsiderable amount of wrangling for GPT5 to stop doing this. So I don't really like the idea that these models have tons of implicit and hidden behavior, to 'soup up' pedestrian prompts.

I've been wanting to make an RTS but found it a bit daunting. I thought this might be a bit challenging for LLMs too but I guess not! This is nice to see.

Very neat! Can't wait for the Sol version!

Here it is: https://senko.net/vibecode-bench/2026/rts-gpt-5.6-sol.html

Clearly much better than the Terra version. I'd say its on par with Fable, and the observed differencies are more due to random luck and open-ended prompt, rather than model capability. (Edit: after some more testing, perhaps not on par - somewhere between Opus and Fable, is a better description).

Fable did better pathfinding and has more terrain variety, visually the map looks better, especially soft edges of the fog of war. And the enemies.

Sol took more care with tiny ux details, added help, and more building varieties.

Could we see the prompts, though?

I did a quick comparison of models a couple of months ago by giving it a MYTXTADV.BAS file and give them all the same prompt to create a sprite-based version of a text adventure game I wrote in Basic over 30 years ago.

It was interesting to see where the approaches were similar and where they diverged.

For local coding agents, I test them by asking them to recreate the BASIC game Taipan (from 1979) in Python. So far one got close, making a terminal version with ASCII menus ala Drug Wars for the TI-83.

Here's the prompt I used:

> Create a simple but functional real time strategy (RTS) game similar to old WarCraft, StarCraft or Command & Conquer games. The player should be able to build buildings, create units, gather resources and should uncover the whole map. No AI or multiplayer needed. Use simple but nice-looking graphics. No sound. Implement everything in HTML/CSS/JS, everything in a single file (you can use 3rd-party js or css libraries/frameworks via CDN).

sort of looks like it was made with mobile in mind but I can't get it to work.

Reply on news.ycombinator.com