Mistral Fine-Tune | Hacker News Reader

Mistral Fine-Tune

github.com

191 pointsalexmolas62 comments

Is finetuning still worth it as models advance so fast? Any real-world use cases?

For example, Bloomberg trained a GPT-3.5 class LLM on their financial data last year and soon after GPT-4-8k outperformed it on nearly all finance tasks.

We ended up focusing on having high-quality eval data and an architecture that makes switching to new models easy.

I think their training methodology was suspect.

Also I think there’s a bunch of important techniques that just are t shared.

Function calling might be one reason. If your app has a lot of custom functions for interacting with tools, fine tuning may be preferred over using context tokens.

Can you recommend a tutorial/document about fine tuning a small model for selecting the correct functions ? Would be great to have something local that can replace OpenAI functions/tools API.

The linked GitHub article above describes syntax for fine tuning Mistral for function calling.

Here is an example of data prepared for fine tuning Llama for function calling...

https://huggingface.co/datasets/mzbac/function-calling-llama...

I'm unaware of any comprehensive guides - we're still in the wild west.

When it comes to traditional NLP related tasks, LLMs are far below dedicated NLP pipelines like POS tagging and feature tagging. However, fine tuning bridges the gap quite a bit between the two.

It's a narrow domain, but so is most of programming. I think if you're just training a general purpose LLM to be more inclined towards your data -- no, fine tuning is probably not very relevant. But if you're trying to solve a very specific yet fuzzy problem, and LLMs can get you _part_ of the way there, fine tuning is likely your best bet.

Can you share a bit more on which tasks are you discussing about?

Not the OP, but you can take a look at https://spacy.io/usage/spacy-101 to get a sense of what traditional NLP tasks look like. These things can be done much faster than LLMs with appropriate tooling (such as spaCy) and they don’t risk hallucination.

Also you can do it at scale with SparkNLP: https://sparknlp.org

Many of the use cases I've seen for LLMs would actually be better with NLP.

Which kinds of use cases?

I’ve tried PII anonymization with standard NLP approaches and LLMs have been (way) better at this task in my experience.

In this comment I was referring to POS tagging and feature extraction.

Another use case for fine tuning we have as well is to reduce a 5-shot prompt that we have to run hundreds of times per request down to a “0-shot” (heavy emphasis on the double quotes.

Run five shot on gpt-4o a couple thousand times, then fine tune on cohere’s command-r or haiku or llama3 8b or whichever small but mighty llm. You can reduce costs by 99%, or somewhere in that ballpark, without really sacrificing quality on 99% of the queries.

I use a programming language that isn’t publicly available and it would be useful to have a model understand it. AI isn’t something I’m super knowledgeable about but fine-tuning sounds like a good way to accomplish that.

Fine tuning can be useful if you need to generate lots of output in a particular format. You can fine-tune on formatted messages, and then the model will generate that automatically. That could save a bunch of tokens explaining the output format in every prompt.

You can use structured generation instead of fiddling with the prompt, which is unreliable. https://github.com/outlines-dev/outlines

Does this Python package control the LLMs using something other than text? Or is the end result still that that Python package wraps your prompt with additional text containing additional instructions that become part of the prompt itself?

Looks like it actually changes how you do token generation to conform to a given context-free grammar. It's a way to structure how you sample from the model rather than a tweak to the prompt, so it's more efficient and guarantees that the output matches the formal grammar.

There's a reference to the paper that describes the method at the bottom of the README: https://arxiv.org/pdf/2307.09702

The output of the LLM is not just one token, but a statistical distribution across all possible output tokens. The tool you use to generate output will sample from this distribution with various techniques, and you can put constraints on it like not being too repetitive. Some of them support getting very specific about the allowed output format, e.g. https://github.com/ggerganov/llama.cpp/blob/master/grammars/... So even if the LLM says that an invalid token is the most likely next token, the tool will never select it for output. It will only sample from valid tokens.

No it limits what tokens the LLM can output. The output is guaranteed to follow the schema.

Quality suffers at very large contexts, and you might want to use your large context for something else (e.g. a long current conversation or a lot of recent data).

Finance is a very mainstream, broad, english-language dominated field. Most people don't work in a field that is so generalist-LLM friendly.

>Is finetuning still worth it as models advance so fast? Any real-world use cases?

Internal corporate data GPT4 was never exposed to?

I still haven't seen really convincing evidence that fine tuning is useful for the internal corporate data use-case (as opposed to RAG which seems to work really well.)

Finetuning would never add any new knowledge. For internal corporate data, use RAG or train a model from scratch. Finetuning would not help anyone answer a question from that data.

How does one do RAG? I see it mentioned like 20 times

you supply to the LLM the subset of data that you need for a specific prompt. That's RAG.

This article reviews some of the most advanced RAG methods: https://medium.com/@krtarunsingh/advanced-rag-techniques-unl...

(not mine)

This isn’t true at all. Fine tune with a single sample and see what happens to your model.

Will share a longer post on this once I finish it. I have tested this multiple times, on bigger models, on custom smaller models, and it does not work.

In a strict sense, finetuning can add new knowledge but for that you need millions of tokens and multiple runs without using LoRA or Peft. For practical purposes, it does not.

Bring on the new post and more details, this stuff is interesting.

PEFT is not for adding knowledge, that's obvious

I get the sense that you want to do the anti-RAG: take some relatively small corpus of data, train a lora, and then magically have a chatbot that knows your stuff...yeah, that will not work.

But chatbots are only one single use-case. And broadly I think this pattern of LLM-as-store-of-knowledge is a bad one (of course, until ASI, and then it isn't).

That said, you absolutely can impart new knowledge through fine-tuning. Millions of tokens is a rather small hurdle to overcome. And if you're not retraining with original/general data, then your model will become very specialized and possibly overfit...which is not an issue in many instances, and may even be desirable.

Yes.

I have non-English human data annotated in a format that was designed for a very specific health-related study. LLMs haver never seen these annotations, non-English LLMs are not a top priority for companies, and we can only use offline-first ones for data privacy reasons.

In this scenario fine-tuning a general purpose LM works wonders.

What is your strategy for this? Do you finetune when a new flagship model is made available? You said local first, so I'm guessing you might have finetuned llama. But there are llama fine-tunes available which have better performance than the base model. How do you choose?

Our strategy is to take a well-known, battle-tested model as a base, train it, and then hopefully one day release the fine-tuned model on HuggingFace.

Other than that, fine-tunes don't really matter for us because not many people are rushing to beat the top models on (say) Georgian POS tagging or Urdu sentiment analysis.

As long as the model can turn language into a reasonable vector, we're happy with it.

Fine tuning is generally not worth it over RAG with newer models unless you have a niche data set that's very different from the pretraining data. LoRAs/control vectors are also generally better as they don't induce forgetting or hallucinations, and they're usually "good enough" with RAG.

My sense is that finetuning might still have a role if your use is high-volume and with a narrow/specific goal. For example, we have GPT based summaries of our customer contact calls. It's fairly high volume (a typical bank will handle millions of calls a year).

We're considering finetuning for 2 reasons:

(1) the current system prompt with instructions is getting quite large* and we see more challenges for GPT to stick to the instructions. We could finetune with our historic summaries and a simplified prompt to slightly improve the performance of the summaries / lower token count (performance). The idea would be to then continue improving our system prompt again from that new starting point. Exploration still to be done though.

(2) we might be able to finetune a much smaller model to do what the larger model is currently doing (cost, sustainability)

* The larger instruction prompt is because there are lots of specific needs for the summaries (e.g. how to write the summary, what to exclude (health, etc.), what to include (actions taken), and we e.g. give examples of good vs bad summaries to improve performance.

I think the bitter lesson kicks in shortly after you complete fine tuning, and openai release their next model which performs better at the same task.

No fine tuning seems pointless right now. you can't even fine tune the frontier models.

My startup has data the Internet doesn't have.

I want it to be fixing knowledge based on my data and fine-tune it to be adjusted to my use case already.

I also want a.feedbackloop and can't start to send more and more context with the payload just because my feedback loop adds / finetunes data

This seems to be the narrative Microsoft was pushing at the Build conference this week.

Weird, because they make one of the best small language models (phi series) which is great for finetuning.

I'd summarize it as prompting for input, finetuning for output.

RAG is a far better option for making a model work with your data/information.

Finetuning is better for making it output in a language, style, data format, programming language, etc.

not all domains will receive enough representation in training datasets, due to interest and/or access.

in the current landscape, specialized, smaller systems will still be the most efficient way ahead in the near future.

>ps, but for smaller models, such as the 7B a single GPU suffices

If I wanna do this, what GPU would I need?? I have a 3060 TI (laptop one) and i9 with 16 gigs of ram. I don't have AWS quota, or quota in GCP either. have heard of paperspace, but I want to quickly get started with fine tuning Mistral, cause we are planning to use some of their models for an engagement I am working on.

Check out this site: https://www.hardware-corner.net/llm-database/Mistral/

They list hardware requirements by model and you can select your VRAM and system memory to filter available models.

Unless I'm missing something - this is for inference only, using quantized models. Fine-tuning usually has much higher requirements.

Unless you have zero budget, I would strongly suggest going up to a "gaming" desktop computer. A gaming GPU can put out 300 W of heat no problem. If your laptop GPU did the same it would melt, it's going to be limited to around 100 W. And heat dissipation is directly proportionaly to speed.

Plus in a desktop, you have the ability to upgrade to faster GPU or even go for multiple GPUs.

Caveat: such machines, especially multi-GPU, are loud and produce enough heat to warm up a whole room quickly.

And cloud will likely be cheaper if you believe you will not spend more than 10% of the next couple of years running the GPU at full blast.

>heat dissipation is directly proportionaly to speed.

Power limiting Nvidia GPUs up to around 60% of max TDP can offer a substantial reduction to heat and noise with minimal performance loss, but agreed on everything else.

I'm using an M1 MBA and was considering getting an eGPU instead of buying a PC, would you advise against it? What variables would you consider?

There are no Nvidia drivers for Mac.

Try out openpipe. I am working with it at my current company and have seen some good results.

You can get a hetzner gpu box for 184€ a month. My company has been fine tuning mistal and llama3 on the rtx4000 these have. It is slightly limited in ram at only 20gb but we found dropping the quantisation has helped with larger input token counts.

They also now do hourly rental.

It will be very interesting to see which tools become the de-facto for each of the common LLM use cases.

The landscape is so fractured, I feel like I haven't even heard of most tools. I encountered Microsoft's olive the other day and it was completely new to me.

Now that many open source LLMs are already “useful” it is key to ease the development _around_ them, in particular in a way where users-developers can leverage on “private data” (really, data that is not present in the pre-training of these models).

Even if in the repo it is stated that it is optimized for big models (needing A100/H100), I still feel this can help with smaller models more than large ones.

We can extend “If you build it, they will come” with “If you provide the tools, the will build”

> If you provide the tools, the will build Only makes sense if the incentive of learning that tech provides expectations of future benefits.

there is also the joy of experimenting a “new and revolutionary” technology. Although yes, with big tech and big salaries there is an economic incentive

The "weights" bit is interesting. HuggingFace's SFTTrainer allows you to train on completions only if you want, but while that makes sense for us humans, LLMs are generally better of learning to predict the entire input. This allows you to get the best of both worlds.

Can this be optimized so training of larger variants is possible with a pair of 3090 or 4090?

This would involve a lot of effort, but seems to be doable. Here's a starting point that discusses some options: https://huggingface.co/blog/trl-peft

How can i train my Whats app chat model ??

What does this mean? How can you train a model on your own WhatsApp messages? To what end? To write like you? For RAG Q&A?

Reply on news.ycombinator.com