Brood War Bench
bw.swerdlow.dev
bw.swerdlow.dev
https://web.archive.org/web/20091124210529/http://eis.ucsc.e...
There's a great contemporary Ars Technica piece by a competitor:
https://arstechnica.com/gaming/2011/01/skynet-meets-the-swar...
As an undergrad I did a project using genetic programming. It was not very successful, but it was a lot of fun.
https://tomisin.space/archive/starcraft-genetic-programming/
Unfortunately the proxybot limitations precluded me from expanding its capabilities so I tried to switch to having ruby embedded in C++ and I basically got mired there and was eventually distracted by real world concerns like actually finishing my degree.
I think I played against Krasi0's bot a couple times in the early days. Hopefully they stuck with AI and have suddenly become crazy rich after 2016. It certainly wasn't a given that AI was going to lead to a fruitful career back then, let alone to unimaginable riches.
https://theaccidentalengineer.com/adversarial-machine-learni...
I was wondering how you could possibly sync up something like a Carrier that has multiple fighters.
Same way as everything else - you play out the simulation and see what happens. The behavior of the interceptors is as deterministic as any other unit. (They just happen to not be directly commandable by their owner.)
The key expression for playlists or yt browsing is this `AI 업스케일`
f.e. https://www.youtube.com/playlist?list=PLWQeRMoEALvp75uolIpIE...
I love StarCraft. I started playing it right from the beginning, most of my friends right now are from that era. I literally met people that have spread to almost every continent when I was in my early teens. We played at internet cafes and did not have access to the internet, that was priced differently...
I miss those days so much.
Everybody was from a different background back then, and nobody was anything other than a guy that plays StaCraft at the cybercafe... And now, we are in our 40's and I know Math teachers, history teachers, oil rig operators, software programmers, professional gamers, lawyers and more... hahah So crazy to think about it... and I know them, we talk, what a world.
My first time playing StarCraft was at summer camp around a decade after it came out.
All the smartest people played it so I wanted to too. Great decision, I have been continually impressed with the people who StarCraft introduced me to.
We were all teenagers having fun though. The friendships you speak of never really took place for me. I wish they did, because it was always jarring to unplug and get back to the real world (school, uni, work) and associate with people who had no idea of my other parallel life online.
I've watched several replays on this. It using map hacks is lame but it also doesn't always take advantage of them. It sometimes does respect its own fog of war. It mostly wins with incredible micro (kinda has to, it's macro kinda sucks).
Most of its few losses come from drops (it never makes turrets and doesn't know how to handle them), bad macro (blocking its own ramp) or just incredibly unconventional play from its opponent (which is certainly not in its training set).
"Oh it's able to issue commands too fast." "Oh no, I give it full map access and it uses that."
Ego shooters are, naturally, also not good benchmarks.
OpenAI won DotA2 in 2019, a way better game.
I don’t applaud people using aim assist in CS for the same reason. We literally don’t have access to the same tools. That’s why chess has been such a great target for years. One’s ability to manipulate the pieces has no bearing.
That's exactly why it's a bad game for human/AI competition.
Protoss: powerful and expensive frontier coding agents you directly micromanage for the toughest tasks
Terran: versatile team comps of dedicated agent roles you can delegate well-defined tasks to
Zerg: massive swarms of specialist custom agents inside your apps that you evolve and optimise for speed and cost
Knowing every faction has its strengths and weaknesses helps me decide which tools to use for the job.
Programming an engine OTOH is a skill that is more general and they should all have.
Might be useful to have the target of the engine be some specific virtual machine that gets a strict cycle budget-- e.g. execution runs so many cycles, and result is read out of a specific memory address at the end (or when it terminates early).
And it's also just bencmaxxing bait: you can get a huge improvement on the task by RLing on it, but make no improvement on anything else. Doing so would just waste model capacity.
If you could tell that every LLM was equally not being exposed to the task then you could justify it as a test of abstract reasoning, but you can't. So it ends up on how much go transcripts ended up in the training, which is ... not a very interesting metric.
(I have not reached single digit kyu)
I will always love this and now I'm going to play it again. Remastered and on a fancy modern machine.
It sounds like it might have been actually played in real time, which would be very important to distinguish.
I have recently seen other harnesses letting agents play real-time games in what seems like discrete time slices, turning eg Portal into something turn-based https://www.youtube.com/watch?v=ruuGXFAmiOE
I’ve scrolled the article, but haven’t noticed any remarks about Fable’s levels.
I'm often asking myself is it better to use higher or lower effort levels, or to maybe drop down to a "dumber" but faster model. And so using a real-time based competition as a benchmark could shed some light on this, I think.
In this vein, here are what I would love to see added in this benchmark:
- Include Google's Gemini models. I keep hearing Gemini being praised for its speed, and I would like to see whether that gives it a big enough edge over the bigger but slower models. - How does a Cerebras-accelerated open source model fare against a much larger but much slower frontier model?
I also feel like in general there is a lot of very low-hanging fruit to start benchmarking models across the spectrum of real-time vs batch-style workloads. Perhaps Brood War sits somewhere quite near the "real-time" end of the spectrum, but what about something like a game of speed chess, or a turn-based game with time limits?
I think what I would like to see the most is for someone to come up with a benchmark that supports tuning the "real-timeliness" of the benchmark, and then running a sweep of a model across the whole spectrum. That could get result in real nice graphs with multiple models on the pareto-frontier, varying based on the hosting provider and the model dimensions.
I saw someone recently try to get an agentic system to play Final Fantasy and it did about as well as a Roomba.
If you just mean general LLMs can’t beat humans yet that’s one thing, but it’s not the case that no system can do so.
Brood War was the first video game to be broadcast on TV in Korea. I'm pretty sure it's still going.
Not sure they’re the best option for raiding, but as a high-level orchestrator for choosing content, that sounds pretty great.
EDIT: Nevermind, seems to work now. Watching a local qwen3.8-flash-next play this.
That’s me. I’ve always struggled with real-time games because I need to pause and think. While I excel at chess and board games, I’m just no good at real-time ones. At last I can only manage by sticking to a fixed set of tactics for a game, which minimizes the need for on-the-fly thinking. Seeing current models face the same difficulty leaves me with mixed feelings.
Must recently I built out a MMORPG puzzle box thing, I wrote a general game architecture doc but left the specific puzzle design up to Fable. Nobody is actively playing rn but I left it up at bot.willmorrison.net.
For agent harness I did Claude Code, Codex, Grok Build. This was primarily a cost driven decision — I have a lot of free tokens and I didn't want to pay API prices for this.
For game harness I used minimal BW-API issue command and get observation apis as tools. I felt this was the most fair way to do it on my small scale.
In the future I would like to integrate code mode and multiple games/I think if it was a best of 5 where each agent could learn from its past games and build its own automations over time that would be much more interesting.
Or possibly even whether they can learn from a game themselves. "Analyse your game for your failures" -> Then give a fresh agent of the same model that "learnings" doc for the next match. Do the rankings change over time, if models can write instructions for future selves?
Although, maybe benchmarking an agent on "how well can you command a swarm to annihilate the Terrans" is how it all starts going downhill...
Surprised the outcomes are so poor though. I recall years ago AI was capable of beating pro level DOTA teams.
I guess in one case it was specifically trained on the interface & game while here it was not?
The film Shrek 3 (2007) took 20 million CPU hours of render time. Games push 60 frames a second. For a visual comparison, check Call of Duty world at war (2008).
Also compare to browsers, which can sometimes scroll smoothly through some styled rectangles and text, and consume gigabytes of ram if you have a few tabs open.
Games have directional sound effects and soundtracks. Don't need 800 Spotify engineers to pull that off.
Multiplayer games solve crazy distributed system problems, making it feel like 'now' when players shoot each other, even with historical latencies of 100-200ms.
AI (in terms of LLMs) seems to be a continuation of that. You used to be able to play 7 AIs on 1998 hardware, at a distinctly "non-beginner level".
That barrel you shoot, is really half a barrel when you're up close, a flat rectangle when you're far, a point-with-mass + a vector for purposes of physics, and not even there for purposes of AI because pathfinding uses a precomputed graph of nodes that's carefully aligned with the map so you don't notice the enemies can noclip through everything other than floors and walls. Etc.
And yes, many games would have scripted enemies or other events come out at you so you don't linger in particular areas too long, lest you spot some of the shortcuts they made.
I grew up wanting to make games, spent my teenage years in hobbyist gamedev communities, and to date, this remains to me the most enjoyable and pure form of exercising software development skills.
If you could win as Protoss by building Dark Archons, raiding your Zerg opponent's base to capture a worker, building a bunch of Zerg production and tech buildings, and attacking with a combined army... you most likely outclass your opponent, and could have won much faster using a more conventional strategy.
So there was enough time. The big risk was that the map would max before you could fully develop the Zerg side.
If you want 6 carriers at the same time, build 6 starports.
If you want 6 ultralisks or 6 mutas at the same time, build 2 hatches (which you already have)
One consequence of this is that changing army composition as Z is very easy.
Like if a model can’t handle a task it hasn’t been trained on extensively, that’s not intelligence it’s memorization.
Really looking forward to playing- are you all limiting boxes?
Marine staggering for example seems like an ideal code mode task.
On the other hand, I do think LLM-based bots will quickly outperform the decision-making of many of the hand-coded bots, so maybe they won't need so much APM to be competitive.
AlphaStar won a showmatch against TLO. TLO was never one of the strongest players in the world. He had been retired for over three years by the time of the match. Google set the rule that their system would have human-like mechanics, but it played several times faster than any human, never issued a wasted action, had an inhumanly fast reaction time, issued commands with perfect accuracy using an API, and could see the entire map at once.
It was later released to the open ladder with more human-level mechanics. Even strong amateurs regularly beat it. I have beaten it myself. It was strong, but not even close to the level of the strongest human players. It had obvious and easily-exploitable deficiencies in strategy and building placement.
I think even the cheater version would have lost handily to Serral or any of the strongest players.
(It apparently beat MaNa as well as TLO, but those matches were never released to my knowledge. I see no reason to assume Google cheated less flagrantly in private than they did in public.)