Hacker News Reader

Comment by superkuh | Hacker News Reader

This might be true for the type of business and institutional uses that can operate under the extremely puritanical filters that are bolted onto gpt3.5-turbo. But for most human person uses the earlier text completion models like gtp3 davinci are incomparibly better and more responsive. But also 10x as pricey. Still, it's worth it compared to the lackluster and recalcitrant non-output of gpt3.5-turbo.

I think over the next couple months most human people will switch away from gpt3.5-turbo in openai's cloud to self-hosted LLM weights quantized to run on consumer GPU (and even CPU), even if they're not quite as smart.

View on HN

A couple months might be too soon imho. But I hope that in 2-3 years there will be a model with similar performance but much smaller size, small enough to run incredibly fast inference + training on my laptop. OpenAI might need to rethink their moat in case that happens.

Think about all the smart ML researchers in academia. They can't afford training large models on large datasets, and their decades of work is made obsolete by OpenAI's bruteforce approach. They've got all the motivation in the world to work on smaller models.

I actually don't think that we will make significant advancements in reducing model size before we make significant advances in increasing available power and compute.

One reason is that the pressure is still on for models to be bigger and more power hungry, as many believe compute will continue to be the deciding factor in model performance for some time. It's not a coincidence that OpenAI's CEO, Sam Altman, also runs a fusion energy r&d company.

But processing hardware has been seeing diminishing returns for years. My CPU from 2013 is still doing what I need; a 1993 processor in 2003 would have been useless.

Where do you see hardware improvements coming from?

Specifically in AI there is huge room for improvement with things like optical computing. AI processing doesn't need to be completely deterministic, as shown by the fact that we are quantizing llama down to 4 bit without too much of a drop in performance. Once you drop that requirement you open the door to using much, much more efficient analog circuits. How do I invest in optical computing...

Can’t ChatGPT design better hardware ?

No but maybe GPT-5 will be able to

Only Deep Thought can do that. And the answer is still 42.

Good to know!

Training these nets mostly occurs on GPUs. CPUs are often hamstrung by their serial performance (e.g. operations that depend on the output of a previous operation end up stalling instruction pipelines.) GPUs still have a decent amount of room to parallelize and maximize compute.

LLaMA-13B 4bit is around 7GB and beats GPT-3.5 175B in benchmarks.

It runs at acceptable speeds on a Macbook Air M1 or a $100 consumer video card.

There is a drop in performance after quantization, so initial benchmarks aren't relevant. Here are some benchmarks for quantized https://github.com/qwopqwop200/GPTQ-for-LLaMa

llm-nasty will find a way.

Stable Diffusion broke free of the shackles and was pushed further than DALL-E could have ever hoped for.

Just wait. People's desires for LLMs to say spicy things and not be controlled by a single party will make this happen yet again. And they'll be more efficient and powerful. Half the research happening is from "waifu" groups anyway, and they'll stop at nothing.

Exactly, there is so much money to be made by generating porn that it’ll be done by this year.

seriously. entertain the humans.

VCR, cable, internet, usenet, web, streaming, VR...

There are so many technologies that were propelled forward because of it, not in spite of it.

Twitter, Reddit, Tumblr...

Tumblr learned a hard lesson when they tried to walk away.

Feeding the AI lots of porn is one way to fix broken fingers.

Its actually good for most human person uses too like writing or learning. I’ve never encountered it refusing to do a task in my actual work.

I would guess the risk to their brand vs the number of actual applications of the unfiltered ai makes it an obvious trade off.

I mean who turns off google safe search when writing an essay or lyrics?

Wait, there are people that aren't children that actually have SafeSearch turned on as more than an accident? Not trying to be insulting, I just genuinely have it turned off in my settings and haven't noticed any of my search results being particularly NSFW and assumed everyone else did too.

If the default is on, most people would have it on.

If the default is off, most will have it off.

If the default is on, most will have it on.

All of which to say, no one cares, and google very likely knows that. Google will only care if enough of their users care. And they will probably operate in a fashion that keeps the maximum number of their users in the "don't care" camp. It's just business.

The default is moderate, which seems to mean "filters irrelevant porn from innocent queries"

>I mean who turns off google safe search when writing an essay or lyrics?

Adults?

I have a hard time imagining anything that comes even close to ChatGPT being able to run on consumer hardware in the next couple of years

That's what I thought 2 weeks ago. I figured it'd be ~5 years before I could do anything at home. But already people have the leaked facebook llama weights running on CPU w/under 32 GB of system ram doing a token a second or so.

that llama is likely much smaller than chatgtp

The whole point of the LLaMa paper is that large models are undertrained and oversized.

not sure where did you get it, but they trained 65b llama too, which outperformed llama 7b on their benchmark.

Here is the paper and synopsis:

https://arxiv.org/abs/2302.13971

> We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

what's your point? If you read actual paper you will see that 65B outperforms that 7B which was discussed above, and size matters.

I don’t think anyone said size doesn’t matter. The point is that by training for longer, models with fewer parameters will perform similarly to GPT-3 models with more parameters.

probably not if gap is so large: 7B vs 175B.

Also, all those benchmark are trash because they can't track data leaks in training data. For example they trained llama on github, where GSM8k eval data is located, of course model will perform well on GSM8K, because it memorized answers.

You don’t have to guess, the information is in the provided synopsis.

I’m sure there are issues similar to your description. Nevertheless, you seem to be a staunch defender of GPT-3, which to me indicates some kind of bias? Like, who cares if LLaMA is better - in fact, isn’t that a good indicator of progress?

> You don’t have to guess, the information is in the provided synopsis.

yes, I checked benchmarks in paper, and there are many where gpt won over 7b llama. Also, it is not clean experiment, because models were trained on different datasets.

> I’m sure there are issues similar to your description. Nevertheless, you seem to be a staunch defender of GPT-3, which to me indicates some kind of bias? Like, who cares if LLaMA is better - in fact, isn’t that a good indicator of progress?

personality rants have been ignored.

:shrugging:

Okay, then.

You can run llama-30b right now on high-end consumer hardware (RTX 3090+) using int4 quantization. With two GPUs, llama-65b is within reach. And even 30b is surprisingly good, although it's clearly not as well trained as ChatGPT specifically for dialog-like task setting.

I can certainly imagine it after seeing https://github.com/ggerganov/llama.cpp

Still a couple years out but moving way faster than I would have expected.

ChatGPT has 175B weights if I'm not mistaken. Llama 7B would not be in any way comparable.

For people who think the number of parameters determines LLM coherence, well, that's a good rule of thumb. But there's an optimal training set data size to parameters count and gpt3 was trained on too little data. The LLM coming out now trained on more data with fewer parameters and achieve something close.

Sure, the 7 billion parameter can't do long outputs. But the 13 billion one is not too bad. They're not a full replacement by any means but for many use cases a local service that is stupider is far preferable to a paid cloud service.

One finding in the LLaMA paper [1] is that our current large models are undertrained. LLaMA with 13B params outperforms GPT-3 175B (not ChatGPT), but an "instruct" version of LLaMA was finetuned over the 65B model and did quite well.

[1] https://arxiv.org/pdf/2302.13971.pdf

It's crazy, but it seems to be happening already. Granted, that's probably still a far-cry from Chat-GPT, but it seems inevitable a few years down the line.

https://news.ycombinator.com/item?id=35100086

I could perceive something like in 1 or less.

M3 Macbook with eGPU functionality restored in conjunction with more efficient programming would mean having enough memory available to all the processors. This would definitely count as consumer hardware.

Custom built GPU-like devices with tons of RAM could become vogue. Kind of like the Nvidia A100 but even more purpose built for running LLMs or whatever models come next.

Moore's law isn't quite beaten yet, so the (hypothetical future) RTX 5090 and 6090 is gonna be insane. Combined with software optimization and refinement of the techniques, along with training != inference, means I think we'll see something better, runnable locally, in a couple of years. The leaps and bounds Stable Diffusion has gone is insane. Facebook's LLaMA is also seeing a similar growth from just having the model available.

I hear that disco music is also experiencing tremendous growth, and is poised to make a comeback any day now.

It doesn't matter that the older model will happily generate text to make your grandmother blush. The usage policy specifically says you can't do that. They even provide an API endpoint for checking whether the input, and output, is allowed or not.

There's nothing stopping you from ignoring it, except for the certainty that OpenAI will simply block you.

Or just keep using davinci because is it’s also really cheap all things considered. I was excited about getting 1/10th the cost but also came to the same conclusion as you as turbo can’t actually do anything. I could care less about getting it to write porn or nazi propaganda but good lord it can’t even write code, do line edits or follow instructions more complicated than simple call/response.

My use case is IRC bots. If you just have the bot responding to only a single line and not knowing any of the chat history, yeah, it can be fairly cheap. But once you try to start giving it short term memory by feeding in the prior ~handful of lines you blow through that $18 of free credit in a couple weeks. Something that costs $25/mo is not cheap for a human person.

I am not happy with your implication that gpt3.5-turbo only doesn't respond to "nazi" stuff and that my users are such people. But I guess getting Godwin'd online isn't new. It literally won't even respond to innocuous questions.

What kinda volume are you pushing though because I also do that and even have it ingest whole pdfs/word docs as conversation context and I get charged like $3/mo on average.

Edit: I’m literally agreeing with you and describing innocuous questions that it doesn’t respond to. I’m saying that if all it refused to do was write hate and erotica it would be fine and I would use it but the filter catches things like code.

With gpt3 davinci we were doing about ~30 requests for text completion per hour (at peak activity) each having ~11 lines of chat history (including up to 190 token responses from davinci) which added up to about ~1000 to 5000 tokens each. So 30*3000 at $0.0200/1000 tokens equals a few dollars per day.

what is an example that you can do with Davinci, but not chatgpt? In my limited experience with prompting you can ask chatgpt to do a lot of things

gpt3.5-turbo fails the turing test due to it's constant butt covering. Davinci can pass for a human. I am speaking only of the API responses. The "chatgpt" web interface is something different.

Reply on news.ycombinator.com