Hacker News Reader

smy20011

Alpaca model leaked again?

No, this is an open reproduction of Alpaca on the leaked LLaMA 7B weight set.

This isn't actually a direct reproduction. It's a bit missleading because it only says it at the bottom. This actually uses LORAs which train way faster and are much more flexible, but they perform a little worse.

There is someone that is doing a direct reproduction of Alpaca in the alpaca repo issues though.

> The weights are based on the published fine-tunes from alpaca-lora

>they perform a little worse.

Be aware that LoRA performs on-par or better than fine-tuning in model quality if trained correctly as the paper shows: https://arxiv.org/abs/2106.09685

Ah thank you for the correction this is good to know! I'll admit I'm not super experienced with them directly.

I think the current LoRAs for Alpaca are reportedly performing a little bit worse - I'm guessing from a lack of optimization so I'm sure it'll improve.

Reply on news.ycombinator.com