Hacker News Reader

Comment by cubefox | Hacker News Reader

Your conclusion seems not to be warranted since you haven't tried out the 65B model.

I agree, but I think my experience is representative. So far most human people don't have the resources to be able to use 65B. And most small companies / university groups don't have the resources to fine-tune a 65B.

I've talked to a couple dozen people in real time who've played with up to 30B but no one I know has the resources to run the 65B at all or fast enough to actually use and get an opinion of. None of the open source llama projects out there are using 65B in practice (despite support for it) so I think my 30B and under conclusions are applicable to the topic the article covers. I'd love to be wrong and I'm excited for this to change in the future.

llama.cpp has great support for 65B, and I've been using it on a Linux box (I upgraded my RAM for that purpose). 64GB of RAM for a desktop is like $160 today, so it's not out of reach for most people here if they cared to.

Admittedly, it's quite slow and therefore not useful for chatting or real-time applications, and it's unreliable enough in its quality that I'd like to be able to iterate faster. Definitely more of a toy at this point, at least when run on CPU.

Could you quantify "quite slow"?

A token per second-ish with a Ryzen 7 5800X. If I run it for too long it gets slower as heat throttling kicks in, I need a better cooling system if I'm going to run it non-stop.

For a bit of comparison, if you've tested, how fast are 13B or 7B on the same setup?

Really fast. I didn't bother timing, but they're faster than ChatGPT by a long shot. I didn't spend very long with them because the quality is so much worse than the 65B.

I should probably go back and try again to see if it's worth it for the extra speed, now that I've played with 65B for a while.

i've had the same experience tbh, 7/13/30 on ryzen (local) and intel (server) both on rhel/centos. It's a shame really

Is this with full or empty context?

Good question--I'm counting it from empty and around empty. By the time it gets to full I'm also getting heat throttling (I can tell looking at the temp), so it's hard to know the degree to which the slowdown is one or the other.

I agree with your premise: I have used 65b variants and of course they’re not as good as OpenAI. GPT3 has 175b parameters, and OpenAI has done more RLHF than anyone else. Why would we expect to get comparable performance with models a fraction of the size and a pittance of the fine tuning?

That said, it’s clear that replicating GPT4+ performance is within the resources of a number of large tech orgs.

And the smaller models can definitely still be useful for tasks.

> Why would we expect to get comparable performance with models a fraction of the size and a pittance of the fine tuning?

LLaMA incorporated new techniques that make 65B perform way better than GPT-3's 175B so the model size argument is not very strong.

It's worth pointing out that size isn't everything. From Meta's benchmarking [1] LLaMA 33B outperforms GPT-3 175B, Gopher 280B, Chinchilla 70B and even matches PaLM 540B on a bunch of common evals. Those interested in doing more comparisons can look at https://crfm.stanford.edu/helm/latest/?group=core_scenarios and https://paperswithcode.com/paper/llama-open-and-efficient-fo... to see where it sits (with some GPT 3.5 and 4 numbers here: https://paperswithcode.com/paper/gpt-4-technical-report-1)

I'd agree the secret sauce for how great the newest services perform is probably in the fine-tuning. We're seeing almost daily releases of fine-tuning data sets, training methods and models (at lower and lower costs) so I'm personally pretty optimistic that we'll be seeing some big improvement in self-hosted LLM performance pretty quickly.

[1] https://ar5iv.labs.arxiv.org/html/2302.13971#:~:text=Table%2....

This is a good point. Even if ordinary people did have the resources to run the 65B well on their existing devices, the speed would limit its usefulness quite a bit. In practice, 30B is what most people are going to interact with (if even, I've seen a lot of projects use 13B).

My experience here is pretty similar. I'm heavily (emotionally at least) invested in models running locally, I refuse to build something around a remote AI that I can only interact with through an API. But I'm not going to pretend that LLaMA has been amazing locally. I really couldn't figure out what to build with it that would be useful.

I'm vaguely hoping that compression actually gets better and that targeted reinforcement/alignment training might change that. GPT can handle a wide range of tasks, but for a smaller AI it wouldn't be too much of a problem to have a much more targeted domain, and at that point maybe the 30B model is actually good enough if it's been refined around a very specific problem domain.

For that to happen, training needs to get more accessible though. Or communities need to start getting together and deciding to build very targeted models and then distributing the weights as "plug-and-play" models you can swap out for different tasks.

And if there's a way to get 65B more accessible, that would be great too.

I'm pretty confident that the landscape is going to look very different by the end of the year, as there are so many people poking around now. I think that significantly smaller models will definitely be good enough for specialized tasks, but an equivalently tuned larger model will always be better, the question is by how much. On Meta's benchmarks [1], there's only a tiny gap between 30B and 65B, for example.

For 65B, GPTQ 4-bit should fit LLaMA 65B into 40GiB of memory. Currently the cheapest way to run that at an acceptable speed would be to use 2 x RTX 3090/4090s (~$2500-3000) or maybe a Jetson Orin 64GB (~$2000). I've seen people trying to run it on an M1 Max and it's just a bit too slow to comfortably use (I get a similar speed to when I try it on my 5950X - about 1-2 tokens/s), but it seems like it's within a factor or two of being fast enough, so not out of the question that it might get there just through software optimizations. I'd definitely upgrade to a 7950X/X3D or a Threadripper (w/ 96GB of DDR5-5200) if I could get 65B running at a comfortable speed all the time.

I think training is also advancing at a pretty good clip. LLaMA-adapter [2] is doing fine tuning of LLaMA 13B on a single 8xA100 system in 1h (so for ~$12 for a spot instance) and was already over 3X faster than Alpaca's training.

To me, the biggest thing limiting easy plug-and-play distribution is actually LLaMA's licensing issues, so maybe someone will offer a better open foundational model soon and the community can standardize on that. It'd be nice to have a larger context window (Flash Attention?) as well.

[1] https://github.com/facebookresearch/llama/blob/main/MODEL_CA...

[2] https://github.com/ZrrSkywalker/LLaMA-Adapter

FYI, many of us are indeed running 65B. I’m running 65B at 4-bit and getting about 7.5 tokens per second. Granted, I have a beefy machine with 2x 3090s and Nvlink but certainly well within the realm of any small lab.

Is it hard to spin up an appropriate EC2 instance with 64GB+ of additional RAM? The m6a.8xlarge seems to have 128GB and costs $1.38 per hour. Was going to try this myself, but now I'm wondering if even that won't be worth the trouble. (I know this is not "your own devices" but would give a feel for what you're missing with the smaller models.)

AWS is notoriously pricy. You should be able to find a much better deal elsewhere.

his conclusion is simultaneously not warranted and correct

a like-for-like comparison would be GPT-4 against the larger models like LLaMA 65B, but those cannot be run on consumer-grade hardware

so one ends up comparing the stuff one can run... against the top stuff from OpenAI running on high-end GPU farms, and this technology clearly benefits a lot still from much larger scale than most people can afford

the great revelation this year is how much does it get better as it get much, much bigger without a clear horizon on where will diminishing returns be hit

but at the same time, some useful stuff can be done on consumer hardware - just not the most impressive stuff

if "leaks" are true then even a 65B model is a toy when GPT-4 is supposed to be 1000B I would really like to have an open source model that could compete with even gpt3, but basically all lammas I tried failed to create correct email validation regex, where gpt3 is able to even refer to what rfc it is

13B on my computer does produce useful output, it's pretty hit and miss though

GPT-3.5 OTOH is much better, but it's also much better at producing convincing-sounding but completely incorrect answers

The 65B model runs fine on a Mac Studio with 64GB of memory.

The output is unremarkable; it’s not significantly better than the 13B model for most uses.

GPT 3.5 is an order of magnitude better at least.

You cannot heavily quantise models and assume they didn't degrade significantly.

To run it properly you need a lot more than a Mac Studio, and then comparisons need to be done more or less seriously, not just a few random prompts, because anything in a black box will "cheat" and will be fine tuned to do well at popular benchmarks.

Reply on news.ycombinator.com