Hacker News Reader

Comment by gorbypark | Hacker News Reader

I’d keep an eye out on huggingface. I’ve yet to see alpaca 65B, but there are copies of 30B (and 7/13B) floating around already. I’m sure someone will eventually pony up for the compute time to do 65B. I’ve found watching these models, https://huggingface.co/decapoda-research and then looking at the “spaces using …” section is a good way to find what’s being worked on (for example, check the comments for ipfs/torrents links). Also just doing a search for “alpaca” could work.

You won’t be able to do the finetuning on your MacBook, but should be able to run inference on a 4 bit quantized model.

View on HN

Is the coding itself pretty trivial, cos I've got a decent pc and fancy giving it a go. Got threadripper 3960x with 128GB RAM and 2x quadro 8000s on nvlink. It's not an AI powerhouse by any means but should be better than a Macbook

I’ve been sidetracked with work but planning on tuning llama 65B to produce alpaca 65B, will distribute via huggingface or torrent..

FWIW running the 30B alpaca-lora model quantized to 4-bit via llama.cpp has given me great results, and while I don’t expect much of an improvement from 65B at FP16, 65B will probably perform better than 30B when quantized

The interesting next steps in my head are more focused around curating a better instruction-tuning dataset using GPT-4, then fine-tuning again, and integrating the LangChain project with the resulting agent

Sounds great! It would be great if you could release your intermediate steps, so Alpaca-65B and then Alpaca-65B-LangChain, etc. I'd like to compare the model sizes directly when fine-tuned using the same method / dataset!

I also just realized that I don't believe there's an "alpaca-native" 30B floating around, just the alpaca-lora one, so 30B would be pretty cool too (and the biggest I can run w/ llama.cpp on my MacBook).

> I’m sure someone will eventually pony up for the compute time to do 65B.

How much would it cost? Can you give a breakdown of hardware/paas requirements and the costs?

I was able to finetune the pretrained 13B model (using the same alpaca dataset, just running on my dime) in about 14 hours on an A6000. I think it will work out to between $50 and $60 when the bill hits me.

Be warned everyone is slapping code together so fast that, if your experience is like mine, you'll spend most of your time working around assumptions made by prior authors or hand merging patches between forks to get your setup running well.

Thank you! Very insightful.

Crazy pace.

Another datapoint, first hand report from a random internet person that 7B took 2.5-3hrs on a 8xA100 80Gb setup. If it ends up being a straight up linear thing, that means about ~27 hours for the 65B model. Depending on the host and if it's a preemptable instance or not, that could be about $12-30 per hour.

Have a link? I haven't seen any finetuning scripts in the wild that train a PEFT model on a multrigpu setup yet and would love to play around with one.

The original Alpaca repo has the training script. The readme has the torchrun command and arguments used for train.py. https://github.com/tatsu-lab/stanford_alpaca/blob/main/train...

Awesome, thank you!

Reply on news.ycombinator.com