Hacker News Reader

Comment by jph00 | Hacker News Reader

For now, it's for finetuning.

The issue of to what degree it might be possible to train a model from scratch using QLoRA is still an open question. The relora paper showed that it can work in some situations, but attempts to scale it up were unsuccessful. The recent DoRA paper perhaps might allow a "re-DoRA" approach to work. If so, that could be combined with quantization to do "re-QDoRA"!

View on HN

The headline and introduction on the linked page say "You can now train a 70b language model at home. We’re releasing an open source system, based on FSDP and QLoRA, that can train a 70b model on two 24GB GPUs."

How does "fine tuning" differ from "training?" Reading the linked article I had assumed I could create my own trained LLM at home with two 24GB GPUs.

You most definitely can, the main difference is that only partial ~2% of the parameters get updated during training. Say you start from a model like llama-70B which already knows english and has some world knowledge based on its pretraining dataset. It might not be ideal for drastic domain shifts, such as adapting a model to learn new languages (which might require a new tokenizer and model embeddings) but still might be possible to some extent.

Thank you for clarifying. I have been wanting to dip my toes into LLMs at home but obviously I have a steep learning curve ahead of me, and would need considerably beefier hardware!

It’s steep but manageable, absolutely go for it. The more people who understand the tech the better.

You can take an existing 70B model and train it to do a more specific task. You're teaching it the task but you're relying on a foundation model for the base understanding of the world/words/etc.

OK, that makes sense. Thank you!

The article actually sneaks in a footnote that answers this (https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html#fn1): "Throughout this article “training” can refer to either pre-training, or fine-tuning".

(Generally, we've told students at fast.ai since 2017 that they should almost never be starting from random weights -- most of the time it's best to start with a pretrained model and fine-tune that, even if it's from a somewhat different domain to the problem you're working on.)

Have you changed your mind on „The End of Finetuning“ (https://www.latent.space/p/fastai ) or did I simply misunderstand that?

Oh, and thanks for quirky stuff like your APL video!

The title of that podcast isn't something I actually said (IIRC). I commented in that interview that I feel we should not consider pre-training and fine-tuning to be as separate as we do now.

So you‘re generally in favor of mixing training data without separating them in phases, but when I use pretrained weights (as you recommend instead of random weights) I generally do not have access to whatever the neural net was pretrained with by someone else, so I have to make do with my finetuning data, yes?

Thank you!

Yes.

"The right way to fine-tune language models... is to actually throw away the idea of fine-tuning. There's no such thing. There's only continued pre-training."

:) i hope i didnt pervert your intent too too much for clickbait or something, i thought it was the spirit of what you said

Digging into the low rank structure of the gradients, instead of the weights seems like a promising direction for training from scratch with less memory requirements: https://twitter.com/AnimaAnandkumar/status/17656138151468933...

Simo linked some older papers with this same idea: https://twitter.com/cloneofsimo/status/1765796493955674286

Reply on news.ycombinator.com