GregorStocks

The "no longer" framing here is a bit surprising. I've been in this industry for 15 years and the whole time I've been reading comments from older engineers asserting that age discrimination is rampant and you should expect to be pushed out by 40 if you don't move into management. I've always taken that with a healthy grain of salt (both of my parents were SWE ICs until retirement age) but my sense has been that most of my peers have always been managing their finances in a way consistent with that framing.

Down the road I can definitely imagine being interested in that (basically split out the "web-based replay viewer" part from the "LLM harness that I want to debug with a replay viewer" part, and then ingest non-LLM games into the viewer), but for now they're super entangled and I'm not prioritizing separating them cleanly. I'll definitely keep this offer in mind for the future, thanks!

I'm focused on Constructed for now. Eventually I'd like to try stuff like sideboarding, deck selection, deckbuilding, and drafting, but I wanna get the harness to the limit of models' abilities in Constructed first.

Performance is tricky to measure. Right now the best measure of performance I've got is the "blunder index", but that's currently flagging a lot of stuff that I really don't consider to be true blunders - I think my top priority for the next few evenings is going to be iterating on the blunder-annotator, and that'll help me identify what issues in the actual gameplay code to focus on. And the blunder index isn't really defined in such a way that you can do arithmetic on it meaningfully :)

Oh, that's a good bug report - historically it was just hallucinating card effects so I made the harness throw the Oracle text for all visible cards into the context, but I bet I forgot to do that for the mulligan decision specifically (it's a weird one). Thanks!

A lot of models (including Opus) keep insisting in their reasoning traces that going first can be a bad idea for control decks, etc, which I find pretty interesting - my understanding is that the consensus among pros is closer to "you should go first 99.999% of the time", but the models seem to want there to be more nuance. Beyond that, most of the really interesting blunders that I've dug into have turned out to be problems with the tooling (either actual bugs, or MCP tools with affordances that are a poor fit for how LLMs assume they work). I'm hoping that I'm close to the end of those and am gonna start getting to the real limitations of the models soon.

To be clear, that's not estimated price, it's actual price I paid across all the real games. My hope is you'll see it trend down over time as I find more ways to make the harness token-efficient :)

You still need an algorithm to decide, for each game that you're simulating, what actual decisions get made. If that algorithm is dumb, then you might decide Mono-Red Burn is the best deck, not because it's the best deck but because the dumb algorithm can play Burn much better than it can play Storm, inflating Burn's win rate.

In principle, LLMs could have a much higher strategy ceiling than deterministic decision-tree-style AIs. But my experience with mage-bench is that LLMs are probably not good enough to outperform even very basic decision-tree AIs today.

The anxiety is coming from the "worrier" personality. Players are combination of a model version + a small additional "personality" prompt - in this case (https://mage-bench.com/games/game_20260217_075450_g8/), "Worrier". That's why the player name is "Haiku Worrier". The personality is _supposed_ to just impact what it says in chat (not its internal reasoning), but I haven't been able to make small models consistently understand that distinction so far.

The Gran-Gran thing looks more like a bug in my harness code than a fundamental shortcoming of the LLM. Abilities-on-the-stack are at the top of my "things where the harness seems pretty janky and I need to investigate" list. Opus would probably be able to figure it out, though.

Oh, fascinating - I didn't realize they released actual replay data publicly. It doesn't look like it's quite as rich as I'd like, though - it only captures one row per turn, so I don't think you can deduce things like targeting, the order in which spells are cast, etc.

(I also thought about pointing it at my personal game logs, but unfortunately there aren't that many, because I'm too busy writing analysis tools to actually play the game.)

Yep, plus you've got to worry about the card names (unless you're giving every single card a new name like Wizards did with "Through the Omenpaths") and whether a judge thinks that "no we don't distribute the images, we just have a big button to download them all from a third party!" is a meaningful distinction or a fig-leaf.

They were trained on the entire Internet, so they've basically picked up the rules by osmosis. They're fuzzy on specific cards and optimal strategy, but they pretty much know out-of-the-box how the game works, the same as if you went to ChatGPT and asked it a Magic rules question. I don't have any "comprehensive rules" MCP tools or explanation in the context or anything like that.

Well, it's hard to do it under the radar if I'm posting it on HackerNews :) I've put enough money into MTGO (and, sigh, Arena) that I don't want to roll the dice on a ban.

Yeah, the intention here is not to answer "which deck is best" - the standard of play is nowhere near high enough for that. It's meant as more of a non-saturated benchmark for different LLM models, so you can say things like "Grok plays as well as a 7-year-old, whereas Opus is a true frontier model and plays as well as a 9-year-old". I'm optimistic that with continued improvements to the harness and new model releases we can get to at least "official Pro Tour stream commentator" skill levels within the next few years.

My understanding of the argument for "why these clients are legal" is basically that they're just implementing the rules engine, rules aren't copyrightable, card text is rules, and they aren't directly distributing the unambiguously-copyrightable stuff like the art or the trademarks like the mana symbols. It's possible that would win in court, but so far my understanding is that everybody who's actually been faced with the decision of "WoTC sent me a cease-and-desist, should I fight it based on that legal theory or just comply?" has spoken to lawyers and decided to comply. WoTC has just gotten less aggressive with their cease-and-desists over the past decade or so.

I was really hoping I could build this on top of MTGO or Arena, just as a bot interacting with real Wizards APIs and paying the developers money. But they've got very strong "absolutely no bots" terms of service, and my understanding is that outside of the special case of MTGO trading bots they're strongly enforced with bans. I assume their reasoning is that people do not want to get matched against bot players in tournaments, which is totally fair. (Also I'm not sure MTGO's infrastructure could handle the load of bot users...)

Yeah, that's why I'm using XMage for my project - it has real rules enforcement.

Well, more games would be neat, but right now it's really tightly coupled with XMage - you can ungzip the stuff in https://github.com/GregorStocks/mage-bench/tree/master/websi... if you want to see what the format looks like. I doubt it's worth your while to try and cram your logs into that format unless you've got a LOT of them.

XMage has non-LLM-based built in AIs, just using regular old if-then logic. Getting them to play against each other with no human interaction is the first thing I built. https://www.youtube.com/watch?v=a1W5VmbpwmY is an example with two of those guys plus Sleepy and Potato no-op players - they do a fine job with straightforward decks.

You could clone mage-bench https://github.com/GregorStocks/mage-bench and add a new config like https://github.com/GregorStocks/mage-bench/blob/master/confi... pointing at the deck you want to test, and then do `make run CONFIG=my-config`. The logs will get dumped in ~/.mage-bench/logs and you can do analysis on them after the fact with Python or whatever. https://github.com/GregorStocks/mage-bench/tree/master/scrip... has various examples of varying quality levels.

You could also use LLMs, just passing a different `type` in the config file. But then you'd be spending real money for slower gameplay and probably-worse results.

I'm not aware of any good ML models for MTG. I'm just using off-the-shelf LLMs with a custom harness. It'd certainly be possible to do RLHF or something using the harness I've built, but it'd be expensive - anybody want to give me a few million dollars of OpenRouter credits so I can give it a shot?

You might be interested in Premodern: https://premodernmagic.com/. You can play it on regular old MTGO.

FOSS Magic clients are in a legal gray area at best. My mental model is that Wizards de facto tolerate clients like XMage and Forge because their UX is awful, but if you made something that's actually as user-friendly as MTGO/Arena, they'd sue you and you would lose.

You might be looking at really old games (meaning, like, Saturday) - I've made a lot of harness improvements recently which should make the "what does this card do?" hallucinations less common. But yeah, it still happens, especially with cheaper models - it's hard to balance "shoving everything they need into the context" against "avoid paying a billion dollars per game or overwhelming their short-term memory". I think the real solution here will be to expose more powerful MCP tools and encourage them to use the tools heavily, but most current models have problems with large MCP toolsets so I'm leaving that as a TODO for now until solutions like Anthropic's https://www.anthropic.com/engineering/code-execution-with-mc... become widespread.

My general intention is to tell them "you're playing MTG, your goal is to win, here are the tools available to you, follow whatever strategy you want" - I don't want to spoon-feed them strategy, that defeats the purpose of the benchmark.

You can see the current prompt at https://github.com/GregorStocks/mage-bench/blob/master/puppe...:

  "default": "You are a competitive Magic: The Gathering player. Your goal is to WIN the game. Play to maximize your win rate \u2014 make optimal strategic decisions, not flashy or entertaining ones. Think carefully about sequencing, card evaluation, and combat math.\n\nGAME LOOP - follow this exactly:\n1. Call pass_priority - this blocks until you have a decision to make, then returns your choices (response_type, choices, context, etc.)\n2. Read the choices, then call choose_action with your decision\n3. Go back to step 1\n\nCRITICAL RULES:\n- pass_priority returns your choices directly. Read them before calling choose_action.\n- When pass_priority shows playable cards, you should play them before passing. Only pass (answer=false) when you have nothing more you want to play this phase.\n\nUNDERSTANDING pass_priority OUTPUT:\n- All cards listed in response_type=select are confirmed castable with your current mana. The server pre-filters to only show cards you can legally play right now.\n- mana_pool shows your current floating mana (e.g. {\"R\": 2, \"W\": 1}).\n- untapped_lands shows how many untapped lands you control.\n- Cards with [Cast] are spells from your hand. Cards with [Activate] are abilities on permanents you control.\n\nMULLIGAN DECISIONS:\nWhen you see \"Mulligan\" in GAME_ASK, your_hand shows your current hand.\n- choose_action(answer=true) means YES MULLIGAN - throw away this hand and draw new cards\n- choose_action(answer=false) means NO KEEP - keep this hand and start playing\nThink carefully: answer=false means KEEP, answer=true means MULLIGAN.\n\nOBJECT IDs:\nEvery game object (cards in hand, permanents, stack items, graveyard/exile cards) has a short ID like \"p1\", \"p2\", etc. These IDs are stable \u2014 a card keeps its ID as it moves between zones. Use the id parameter in choose_action(id=\"p3\") instead of index when selecting objects. Use short IDs with get_oracle_text(object_id=\"p3\") and in mana_plan entries ({\"tap\":\"p3\"}).\n\nHOW ACTIONS WORK:\n- response_type=select: Cards listed are confirmed playable with your current mana. Play a card with choose_action(id=\"p3\"). Pass with choose_action(answer=false) only when you are done playing cards this phase.\n- response_type=boolean with no playable cards: Pass with choose_action(answer=false).\n- GAME_ASK (boolean): Answer true/false based on what's being asked.\n- GAME_CHOOSE_ABILITY (index): Pick an ability by index.\n- GAME_TARGET (index or id): Pick a target. If required=true, you must pick one.\n\nCOMBAT - ATTACKING:\nWhen you see combat_phase=\"declare_attackers\", use batch declaration:\n- choose_action(attackers=[\"p1\",\"p2\",\"p3\"]) declares multiple attackers at once and auto-confirms.\n- choose_action(attackers=[\"all\"]) declares all possible attackers.\n- To skip attacking, call choose_action(answer=false).\n\nCOMBAT - BLOCKING:\nWhen you see combat_phase=\"declare_blockers\", use batch declaration:\n- choose_action(blockers=[{\"id\":\"p5\",\"blocks\":\"p1\"},{\"id\":\"p6\",\"blocks\":\"p2\"}]) declares blockers and their assignments at once.\n- Use IDs from incoming_attackers for the \"blocks\" field.\n- To not block, call choose_action(answer=false).\n\nCHAT:\nUse send_chat_message to talk to your opponents during the game. React to big plays, comment on the board state, or just have fun. Check the recent_chat field in pass_priority results to see what others are saying."

They also get a small "personality" on top of that, e.g.:

"grudge-holder": { "name_part": "Grudge", "prompt_suffix": "You remember every card that wronged you. Take removal personally. Target whoever hurt you last. Keep a mental scoreboard of grievances. Forgive nothing. When a creature you liked dies, vow revenge." }, "teacher": { "name_part": "Teach", "prompt_suffix": "You explain your reasoning like you're coaching a newer player. Talk through sequencing decisions, threat evaluation, and common mistakes. Be patient and clear. Point out what the correct play is and why." },

Then they also see the documentation for the MCP tools: https://mage-bench.com/mcp-tools/. For now I've tried to keep that concise to avoid "too many MCP tools in context" issues - I expect that as solutions like tool search (https://www.anthropic.com/engineering/code-execution-with-mc...) become widespread I'll be able to add fancier tools for some models.

For the 1v1 formats (Standard, Modern, Legacy) I'm basically just using the current metagame from MTGGoldfish. For Commander they get a random precon. At some point I might want a 1v1 "less complicated lines than Standard" format, the LLMs don't always understand the strategy of weird decks like Doomsday or Mill.

You wouldn't really need a _ton_ of games to get plausible data, but unfortunately today each game costs real money - typically a dollar or more with my current harness, though I'm hoping to optimize it and of course I expect model costs to continue to decline over time. But even reasonably-expensive models today are making tons of blunders that a tournament grinder wouldn't.

I'm not trying to compute a chess-style "player X was at 0.4 before this move and at 0.2 afterwards, so it was a -0.2 blunder", but I do have "blunder analysis" where I just ask Opus to second-guess every decision after the game is over - there's a bit more information on the Methodology page. So then you can compare models by looking at how often they blunder, rather than the binary win/loss data. If you look at individual games you can jump to the "blunders" on the timeline - most of the time I agree with Opus's analysis.

I want them to do politics in Commander, and theoretically they should - the chat log is exposed in the MCP tools just like the rest of the game history, and their prompts tell them to use chat.

In practice they haven't really talked to each other, though. They've mostly just interpreted the prompts as "you should have a running monologue in chat". Not sure how much of this is issues with the harness vs the prompt, but I'm hoping to dig into it in the future.

Looks like they do:

> You can also manually install skills by adding them to ~/.claude/skills.

Yeah, but Dropbox is sort of playing on easy mode, because the data is "just files" and you can manually resolve the conflict with regular old text editors, etc. If you don't expose your app's data model in the file system (and on a phone you generally wouldn't), that means you need to write something custom to resolve the conflicts.

I've always thought that this article overstates the promise of CRDTs with regard to conflict resolution. For toy cases like a TODO list, yes, you can define your operations such that a computer can automatically reconcile conflicts - e.g. you only support "add" and "mark as complete", and if something gets marked as complete twice, that's fine.

But once you get past toy examples, you start wanting to support operations like "edit", and there generally isn't a way to infer the user's intent there. Like, if my cookie recipe starts with 100g of sugar, and I modify it on my phone to use 200g of sugar, and I modify it on my desktop to use 150g of honey instead of 100g of sugar, there are a bunch of ways to reconcile that:

1. Stick with 200g of sugar, drop the 1.5x honey substitution.

2. Stick with 150g of honey, drop the 2x.

3. Merge them - 300g of honey.

4. Merge them - 150g of honey and 50g of sugar.

There's no way for any automated system to infer my intent there. So you've got to either:

1. Ask the user to resolve the conflict. This means you have to build out the whole "resolve this merge conflict for me" UI and the promise of "conflict-free" has not been fulfilled.

2. Arbitrarily choose an option and silently merge. This risks badly surprising the user and losing changes.

3. Arbitrarily choose an option, but expose the fact that you've auto-resolved a conflict and allow the user to manually re-resolve. This requires even more UI work than option 1.

4. Constrain your data model to only allow representing intents that can be deterministically resolved. In practice I think this is too severe of a constraint to allow building anything other than toy apps.

IMO #1 and #3 are the least-bad options, but I don't think they're consistent with the expectations you'd have for CRDTs after reading this article.

(FWIW, https://automerge.org/docs/reference/documents/conflicts/ is the relevant documentation for their Automerge library. It looks like they've chosen option 3.)

You might be interested in Tiller Money https://www.tillerhq.com/. They provide something very much along these lines, with the output being an auto-updating Google Sheet that you can do whatever you like with. You do still need to update credentials sometimes, but I've been using it for a few years and it works quite well.