Fine-tuning Mistral 7B on Magic the Gathering Draft

That lines up with my experience- for high-stakes decisions, they rarely give me a great answer. But for low stakes decisions, they do well at giving me a good enough answer. For example, I've been using them to help find gifts for friends and children this month. I don't need the best choice to solve the problem, just a good one.

I wonder if you could define a specific complexity class of problems that LLMs are good at

>I like that this shows how hard even conceptually simple ideas are to achieve in fine-tuning LLMs. Even given a pretty good starting dataset, a decent starting model, etc. this appears to have been a challenge.

Surely, but we can't gloss over the fact that this was accomplished by a single person.

The OpenAI Dota2 experiment produced many interesting behaviours, even the pros are impressed.

> Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Otherwise, you're not measuring the best players on the service. You're training on draft choices where most choices were very good - i.e., win rate sort will show you the luckiest players, not the best ones. That will naturally show up in any validation or testing you do too.

Ahh no just unclear in the post, I'm filtering to players in 17lands with a > 62% match win rate who are drafting at a high ranking (>=diamond rank). I look at all of those players' drafts though, even the ones where they do poorly.

> Your "accuracy" on the draft seems poor. I'm not sure it means what you think it means. Are you saying that when looking at the high win rate choices, where all the choices were mostly good, you happened to pick the choice that isn't the same as the player who originated the data? It actually seems harder to make a choice among all good choices.

Accuracy here is making the same choice from a given pack as one of the good players. Obviously subjective so not a perfect metric, but a decent check on ability to emulate a high-quality drafter.

> did you also try using weighted loss with Axolotl

This is really smart, I didn't think about this! Will add it to my list of things to try, great idea!

> Domain adaptation over subreddits/forums before finetuning may help as well.

I was thinking about this too (along with transcribing draft youtube videos), I'd definitely be curious how much this helps.

> If I'm reading the author's writeup correctly, the prompt he's giving the agent at each pick contains only the names of the cards in its pool so far, and only gives the full text for the cards in the pack it's being passed. It doesn't look like context is being maintained between picks, presumably for context window size reasons.

Not quite -- there's a few ways the model learns the full card text:

* The models are trained on card trivia completions as well, where they're asked to complete the full text of the card as well as information about it (type, CMC, etc.)

* The models do still have to learn next token completion on the cards in packs, meaning they learn to predict the full text of the cards while making draft picks as well.

Net net, the bots learn the text of the new cards pretty comprehensively.

Sparky is the Arena AI, but no one ever accused it of being a good Arena AI - it is very much only there for the new player experience of playing against a dumb computer when you're first exposed to the game and don't know the rules, or for the computer equivalent of "playing against a goldfish" a deck you made to see how it draws or combos. It's not a Chess CPU.

I hadn't seen this, this is awesome! You'd think given the volume of data available that this type of method would outperform an LLM, cool results.

Still some fun things about LLM representations -- you can do fun things like give the bots preferences / personality in a system prompt which is entertaining!

The field is moving so fast, so hard to keep track.

Unless I'm misreading something, it appears that the linked paper appears to use one-hot encoding to represent each of the cards -- not any learned embedding to represent each card -- unless I'm misunderstanding what you mean by "representation learning"?

I tried adding special tokens for a reddit-style dataset once. The format was: `<|post_author|>username<|post_title|>title here...`

The resulting model was so much worse than just formatting everything plaintext. This was with MPT-30B, 15 special tokens, 300M training tokens, and a full finetune.

I may have made a mistake, but I haven't seen any open source finetunes successfully add a large number of tokens yet either.

I was thinking something fairly similar. You could probably do pretty well with a basic NN setup this way, no need for an LLM. It wouldn't work on "never seen before cards" and would probably make some absurd picks when it's wrong, but I'd bet you could get to 90% accuracy.

Without Mistral, how would you get it to generalize to cards it hasn't seen before? I assume by "training a NN to draft without Mistral" you mean where the input layer is just a bitmapped vector of the cards in the pack, right? The killer feature of this experiment is that it works on sets the model has never seen before and has 0 training data on, using just the text of the card. I don't think you can do that without an LLM.

The benefit of the LLMs is that the checkpoint already "understands" a lot by default. Finetuning is relatively cheap and makes many tasks such as this one perform decently well simply by shoving some data into it.

The base checkpoint takes a lot of compute to make, but that's what holds most of it's "knowledge" so to speak.

Making a NN from scratch means you'll have to somehow map the cards into inputs. I have limited knowledge of how MTG works, but most TGG have text descriptions and complex effects. Mapping text to logic is what LLMs are really good at, otherwise you're starting from scratch and will also need a relatively large amount of compute before it starts displaying any type of decent behaviour.

It's also easy for most software devs to do this - finetuning mostly consists of collecting text and feeding it into a finetuning script. You don't need to know linear algebra, what a "convolution" is, etc. to do finetuning.

Can you please point me to good resources on fine tuning? Thanks.

draft data exists, might be a fast alternative

> A question, when GPT-4 contradicts in explanation, how much of them were in fact correct?

It was mostly when a card is good in a vacuum but not as good in a specific set. WOE (which this was trained on) skewed pretty aggressive, so GPT-4 was tended to overvalue strong expensive cards (compared to what good players thought at least).

The longest running fine tuning job took about 8 hours, so ~$5.

I think if you add up all of the learning and testing I did, probably closer to ~$50 total

There is not a lot of great content out there making this clear, but basically all that matters for basic fine tuning is how much VRAM you have -- since the 3090 / 4090 have 24GB VRAM they're both pretty decent fine tuning chips. I think you could probably fine-tune a model up to ~13B parameters on one of them with PEFT (https://github.com/huggingface/peft)

Definitely possible on even older off-the-shelf hardware. I use 24GB 4090s for 13b-sized models and have even used 12GB Titans for 7b models, admittedly at much slower rates.

Would love to compare notes, drop me a email at dshersh at umich dot edu if you'd be interested!

You can get a standard LLM to change tone just by giving it a system prompt/instruction to follow a certain tone.

The only issue there is that sometimes the RLHF seeps through, which can be solved by system prompting even harder.

Aren't a lot of base models fine-tuned with (Q)Lora on instruct-based datasets with good results? I thought this was a very common practice?

Check our Orca. IIRC, it's a technique that aims to encode additional logical capabilities into smaller models by having larger models generate step-by-step solutions to various problems. This doesn't just make them speak more like GPT-4/3.5, but is supposedly making them think more like it as well.

> I was under the assumption that finetuneing LLMs was useful only when you need to change the model's tone (speak like a pirate, voldemort etc).

A lot of why I tried this out was to test the limits of this belief, you see a lot of talk like this out there and it sounded like nonsense to me.

Finetuning is fundamentally not much different than continued pretraining; if you feed the model high-quality and high-volume data I think it's reasonable to expect it to acquire new skills

In order to speak like a pirate, it has to be able to reason :) I've done some fine tunes as well similar to the MTG example, in mine I was fine tuning it to speak JSON and reason about some input- and yes, you can indeed get these models to perform on novel tasks.

Finetuning is a useful workaround for cases when the context size is unsuitable for the task at hand. Anybody knows whether it was ever considered to finetune an LLM on the Linux kernel sources' history and its associated mailing lists?

I've definitely thought about this problem and think it's in the range of 'feasible', but it would be pretty slow and expensive given how much context you need to provide a model for it to be able to reason about the game state. Worth trying though!

> https://www.youtube.com/watch?v=Xq4T44EvPvo

Why would he need to write a game simulator from scratch?

Surely there are OSS versions which are "good enough" ?

(disclosure: not a MtG expert)

17Lands data would definitely help, but humans don't have that data in their head when sitting at a draft table.

Some status (mana cost, stats) of MTG cards are numbers, but the most important part, the effect of the card, is defined in English text. So, in my opinion, this seems to be an area where LLM can work well.

MTGA and MO development team do a lot of work to put card effects into if-then rule, but unfortunately their work is not visible to the players :)

I think a non-LLM model would be good for learning to draft within a particular set, because it can simply encode each card and learn what cards do well together, comparative strength, etc. without learning the stats/rules text on the card.

For learning to draft in one environment and then applying it to a new set of cards, like done here, it would become far more difficult to do without an LLM. The variations and nuances of cards rules text would have to be encoded as features, which would be extremely cumbersome. The LLM gets you some level of understanding of that for free.

High level it's basically: 1. Generate a lot of text examples that look like this: https://gist.githubusercontent.com/davidhershey/f57d0b19563f...

2. The model is effectively trained to predict the next token based on the previous tokens in each of these examples, which has the side effect here of teaching it to make a draft pick based on the contents of a pack.

Nothing too fancy, just next word prediction more or less