Hacker News Reader

Comment by hanselot | Hacker News Reader

So far my experience with Vicunlocked30b has been pleasant. https://huggingface.co/TheBloke/VicUnlocked-30B-LoRA-GGML

Although I haven't had much of my time available for this recently. My recommendation would be to start with https://github.com/oobabooga/text-generation-webui

You will find almost everything you need to know there and on 4chan.org/g/catalog - search for LMG.

View on HN

You should beware that /lmg/ is full of horrible people, discussing horrible things, like most of 4chan. Reddit's r/locallama is much more agreeable. That said, the 4chan thread tends to be more up-to-date. These guys are serious about their ERP.

Reddit is censored, nerfed and full of a different sort of horrible people.

HN is a kind of small miracle in that it's the sort of place where I'm inclined to read the comments first, and seems to be populated with fairly clever people who contribute usefully but not also, at the same time, extreme bigot edgelords and/or groupthinking soy enthusiasts. (Sometimes clever folks who are still wrong, of course, but undeniably an overwhelmingly intelligent bunch.)

This is why I'm aware of /lmg/.

30B models are in no way comparable to GPT-4 even to GPT-3. There is no spacial comprehension in models with less then 125B params (or I had no access to such model). 130B GLM seems to be really interesting as the crowd-source start though, or 176B BLOOMZ, which requires additional training (it is underfitted as hell). BLOOMZ was better then GPT-3.5 for sure, but yeah underfitted ...

How much VRAM does that model need? I'm looking for a 30B byte that I can train a LORA with on an GTX 4090.

Reply on news.ycombinator.com