I’m looking forward to this! Are you using an adapter (I don’t see it mentioned in your article)? I was under the impression you cannot fit 7B at 4 bit since it’ll take 25GB of VRAM or so.
Yes using the qlora adapter that hugging face provides with peft
ahh, I was on my phone before so I must of glimpsed over it, I see it on the last section. Thanks!