Hacker News Reader

Comment by eurekin | Hacker News Reader

This field is in crazy progress mode now. Not long ago it was rent cuda gpu only. Now this... AMD could easily chip away some part of the market, if they released a > 24 GB vram gpu now.

View on HN

I hope they do, and I hope it forces Nvidia to release their own 48GB+ consumer card. 80GB is on my long term wish list as it would allow running a 65B model 8bit quantized. I don’t see local models exceeding ChatGPT performance until we get to a point where folks can run 65B parameter models.

yeah agreed, its sad to me that we're 2 months after llama and "nobody" is seemingly doing any advances of fine tuning on models with more than 7B or 13B parameters.

I have 64gb RAM (not gpu just normal), I’d like to see proof of concepts that the bigger models can be fine tuned and have far more accepted results, or to know if we’re completely going the wrong direction with this

Probably because it’s extremely affordable to train the smaller models.

If I had the gumption (and a data set) I could afford to spend a few hundred bucks to fine tune a model for shits and giggles and I’m just a Random Internet Dude.

I’m all for it, Any Day Now™ I have this idea I want to try and having these people do all this optimization work will probably make it affordable to attempt given I don’t actually know what I’m doing so there will be a whole lot of “yeah, that doesn’t work” going on.

I find, since 30B models are actually quite usable locally if you have good hardware, that I really want something like a Vicuna 30B. That would be amazing. I can only run 65B locally at a speed of 1 token per second, which is too slow to be usable unfortunately.

Wouldn't it be better to get a GPU that shares its memory with the CPU, like in the new macbooks?

Sure, but other than Apple no one has the tight integration needed to make that happen. If Apple leans into their CoreML stuff they have a real opportunity to steal the market from Nvidia.

Long term maybe Nvidia is able to release release some integrated ARM chip, but I’m not holding my breath.

Nvidia sells data center products, Apple is not going to steal this market, they aren't even in it. The gaming market is dominated by Windows, I don't see this moving meaningfully just because apple has some marginally better hardware at this point in time.

The Risc V area, now there we can talk about disruption long term

NVidia Jetson is a line of products with tight integration between CPU and GPU. As just one example.

Does it? I'm not aware of any way to share main memory with the Jetson CUDA cores.

If I'm wrong I'd love a pointer to the docs about it!

How do you imagine consoles work? And what about Apple's tight integration has actually made itself useful? The developer experience is terrible.

It’s useful because you can buy a Macbook with 64GB of RAM, and then use that RAM as VRAM to run your LLM.

Unless you're doing training, is there much point in 8-bit for this model size? My understanding is that the larger the model, the less affected it is by quantization; for 65b, 4-bit gives you ~2% perplexity penalty over 8-bit.

I don’t buy into perplexity as a good benchmark for the usefulness of an LLM. In my experience playing with many LLaMA models of various sizes and quantization levels, the higher bit models perform significantly better on complex questions.

Perplexity is a rough metric for sure, but the non-linear dependency is also directly observable. I would definitely agree with 7b and 13b giving better results with 8-bit, but the difference with 30b is much more subtle.

It should also be noted that the method of quantization makes a big difference. In particular, if you were experimenting with llama.cpp, their original take on it was considerably inferior to GPTQ. And for the latter, parameters such as group size can also make a difference.

Question from a noob: how good would it be to run those on a computer with AMD APU (for example Ryzen 9 7940HS) with 128GB RAM and setting aside 64GB for iGPU?

Another noob here. If I had to guess, it's because current models are mostly memory bound. The AI learning gpus (A100, H100 etc.) are not the best TFlop performers, but they have most vram. It seems that researchers found a sweet spot for neural network architectures that perform good on similar configurations, i.e. near real time (reading speed in LLMs). Once you bring those models to cpu, they might get performance bound again. Llama.cpp somehow illustrates that a bit, for bigger models you tend to wait a lot for the answer. I suspect the story would be similar with igpus

So possibly some basic iGPU (maybe even Intel) with lots of VRAM assigned could be enough?

You can try out the demo and benchmark yourself

As long as there is a Vulkan SDK for your AMD APU (likely there is), MLC-LLM can use TVM Unity to generate code for it

Reply on news.ycombinator.com