Hacker News Reader

Comment by haolez | Hacker News Reader

Yesterday, while using ChatGPT-4, it gave me a very long answer almost instantly. It felt like I was using ChatGPT-3.5, including the poor quality of the answer. In the following prompts, it became slow again, as GPT-4 is supposed to be. The quality improved as well.

I think they are trying some aggressive customization on their infra to try to make it economically viable, but it's just speculation at this point.

View on HN

If their statement is that they haven't changed the weights, there still an immense number of things they can toy with.

For example, the prompt may add more 'safety' language in it, which can cause strange differences to occur, or typical_p sampling values, top_p, top_k, etc, or if they do use mixture of experts, they may even be able to 'use less experts' and only run 1/4 of the models, such that the speed is improved greatly.

There's plenty of ways to make the model change output without retraining.

My personal suspicion is that there is some kind of grid/beam search at the very end. You can kind of see (imagine?) it as output speed is not constant.

So you crank down the iterations and voila. Same weights, poorer output.

I heard GPT-4 described as "eight GPT-3's in a trenchcoat" but I'm not sure how accurate that is.

Several people hinted/remarked (starting w/ George Hotz, then others more closely linked to OpenAI) that it's a Mixture of Experts* approach comprised of 8 220B parameter models.

* https://arxiv.org/pdf/2101.03961.pdf

I wonder if additional layers of factories of factories approach can continue to improve it.

I'm not familiar enough with the technology but could it be possible to create a prompt, or multiple prompts to stitch together 8 similtaneous calls to GPT3.5 pulled together and see if the quality is similar to GPT4?

I think the experts part implies that each "GPT-3" has been specialized in some way?

They're not 3.5 models, which are 175B param, they're 220B and the idea is the fine tuning is different for each which is where the 'expert' comes in.

Ah, that makes sense, thanks for clarifying

The mixture of experts approach was found to be inferior than a straightforward transformer. However, researchers discovered that MoE models give substantially better results when guided by fine-tuning. My guess is that GPT-4 was an experiment to prove out that theory at huge scale and I also suspect that its performance astonished OpenAI as much as everyone else.

If it was an experiment I'd like to know why I'm still paying for access to something claiming to be it.

Because you are getting value from it. Why else would you pay?

Because you signed up to pay for a beta service???? You can stop paying at any time.

If you're paying for it, it's a released product, not a beta. No matter how much a company wants to claim it's a "beta", it's simply not. The company just wants to be able to adhere to lower standards.

OpenAI is hardly the only company that pulls these shenanigans, too.

No, beta has nothing to do with if it's paid or not. Lots of products are released, and you pay for them, with early access/ beta.

No one lied to you. No one tricked you. It is in beta. You chose to pay for it.

I never said anyone lied to or tricked anyone. I'm saying that "beta" is a term referring to pre-release testing. If people are paying for it, it's been released and is therefore no longer a beta.

Calling it a "beta" at that point is just pure PR.

> If people are paying for it, it's been released and is therefore no longer a beta.

You can define "beta" whatever weird way you want but don't complain when it's not how literally everyone else uses it and don't complain when you're paying for something that uses the term the way everyone else does

My "weird way" is old-school, I admit, but it's not weird. It used to be mainstream. Google led the charge to deprive "beta" of actual meaning. I won't stop pushing back on it, because the redefinition makes it more difficult to talk about technical issues.

> don't complain when you're paying for something that uses the term the way everyone else does

I won't, because I won't pay to be a beta tester in the first place. I don't care what others do.

> because I won't pay to be a beta tester in the first place.

Apparently you will, because you started this thread with "why am I paying for this beta software?".

this is just not at all true -- companies have private betas that they monetize all the time. yes, the company provides lower guarantees on the product being supported. but the customer gets to use a product far before they normally would if they waited for general availability. that's the core of the idea behind a beta, not related to if it's paid or not

As I said, doing this is not rare and is certainly not limited to OpenAI. But it's misusing the term "beta" pretty significantly.

It's actually 16 GPT 3.5s in a trenchcoat where each one is slightly different, like the minions.

Can anyone elaborate on the specialization? Does it have to do with segmenting the dataset, or with training on different kinds of tasks?

Fine tuning to a specific more constrained set of tasks is my understanding.

I wonder if maybe they cache and re-serve answers to similar prompts? Probably they could use a GPT-4 response as a template and have GPT-3.5 tweak it to be specific to your prompt

No surprises here. The entire architecture is based on mass production. There is an immutable base model and the only mutable part is the chat context. There is fine tuning but it takes a long time and there is no temporal relation between the fine tuning prompts. Even then it costs a ton of money. This immutable model is shared among all the users of their services. There has to be a supercomputer version of this product that focuses on quality over quantity.

Reply on news.ycombinator.com