Hacker News Reader

Comment by r3trohack3r | Hacker News Reader

> Developers should be mindful that Stable LM 3B is a base model. That means it needs to be adjusted for safe performance in specific applications, such as a chat interface. Depending on their use case, developers must evaluate and fine-tune the model before deployment.

I don't know how to fine tune an LLM. Does anyone have good resources on how to do this?

View on HN

Huggingface tutorials are a decent starting point.

The HF toolchain is pretty mature and most llm finetuning projects are a wrapper around HF models, HF Trainer and some config templates.

An LLM by default would be trained like in the example below, but it would take a lot of VRAM and time.

https://huggingface.co/docs/transformers/training

That’s where things like PEFT LoRa, gptq and accelerate contribute to make your training faster/require less VRAM so you could do it on a consumer GPU with 16-24GB.

For example: https://huggingface.co/docs/peft/quicktour

Then for the tips and tricks, either reddit as suggested by a sibling comment, or Discord communities. Huggingface, EleutherAI and LAION discord servers are all great and have super helpful, friendly and knowledgeable people.

/r/LocalLLaMA/ is a good resource.

/r/LocalLLaMA/ is such an interesting mix of academia/researchers (it was called out in a recent paper, regarding context length, IIRC) and odd anarcho-futurist weirdos. And I kind of love it for that.

Yup kinda best subreddit at the moment weird mix of llm training tips and tricks and people who wanna make sex bots.

Ha Stable Diffusion is very similar I notice. People understanding and thriving in new techniques and advanced workflows, mixed in with people who want to generate boobies in new ways.

I found they can be very unappreciative of their data sources for training. Any mention that scraping art from art sites like DA without their consent was unethical is met with arguments about how its exactly the same as a human looking at art and replicating the style.

They refuse to even acknowledge why those people might be annoyed their work was used without their consent or compensation to put them out of work.

you know what you’re walking into there, lawlessone

For getting started, I'd recommend https://huggingface.co/autotrain - you should be able to find docs on the CLI which can let you do your first training run w/ a single command (which includes automatically pulling the model and training dataset from Huggingface). As long as you have an Nvidia GPU w/ enough VRAM should take only a few hours to tune a 3B model on a small dataset like Alpaca.

I’m developing a service just for that. It’s called https://useftn.com and it’ll fine tune models off your json formatted dataset.

I'm surprised by your pricing. $1.20/hr for an A100? That's significantly cheaper than something like runpod. Are you offering this service at a loss?

Which dataset formats are you planning to support? (OpenAI, ShareGPT, Alpaca, Context-based ...)

Same as OpenAI, where it's either a json or jsonl file with {"input": "text", "output": "text"}

Reply on news.ycombinator.com