Hacker News Reader

Comment by m3kw9 | Hacker News Reader

ChatGPT 3.5 is the base level people expect LLMs to be, it would be 2-3 generation(3-4 years) of hardware before we can reach that. Anything below is just going to get bad reviews

View on HN

A small model might be useful for e.g. NPC interactions in a Quest game

3-4 years to run it on your phone seems generous, barring algorithmic breakthroughs.

If I can run 100B+ models on my high-end desktop in 3-4 years I will be very happy.

What do you consider a high end desktop now?

He means a videocard with 512gb of memory or more.

Is 512gb a typo? The current biggest consumer card has 24GB, so we're probably 15 years from a 512GB card (judging from the increase of 4Gb to 24GB between 2012 and 2022).

That's why 3-4 years would be impressive

With 4-bit quantization the requirements are more like 64gb.

I expect we'll see more unified memory designs like Apple's 128GB M1 Ultra.

I doubt it to be honest, desktop GPUs use too much power (and hence produce too much heat) to be integrated in that fashion, and any kind of shared memory will be too high latency.

There are 'desktop' (well server) cpus with 64GB of HBM memory per socket now. And big LLMs can be run on lower memory bandwidth systems (like zen4 chips with 12x ddr5 per socket) at lower performance, but where 1-2TB of ram is no big deal.

But for what applications? Sure, for answering free-form questions I expect GPT-3.5+ quality. I don't think GPT-3.5 is necessary to provide auto-complete in your email client.

Llama 65B finetunes already exceed it in some niches, like roleplaying or specific (coding and spoken) languages

Isn't that the same LLM that doesn't know how many e's are in "ketchup"? Nice.

Aren't you the same user that doesn't understand word-level tokenization?

You must have me confused with an LLM bot that does know how many e's are in "ketchup".

Reply on news.ycombinator.com