Hacker News Reader

Comment by andai | Hacker News Reader

I heard GPT-4 described as "eight GPT-3's in a trenchcoat" but I'm not sure how accurate that is.

Several people hinted/remarked (starting w/ George Hotz, then others more closely linked to OpenAI) that it's a Mixture of Experts* approach comprised of 8 220B parameter models.

* https://arxiv.org/pdf/2101.03961.pdf

I wonder if additional layers of factories of factories approach can continue to improve it.

I'm not familiar enough with the technology but could it be possible to create a prompt, or multiple prompts to stitch together 8 similtaneous calls to GPT3.5 pulled together and see if the quality is similar to GPT4?

I think the experts part implies that each "GPT-3" has been specialized in some way?

They're not 3.5 models, which are 175B param, they're 220B and the idea is the fine tuning is different for each which is where the 'expert' comes in.

Ah, that makes sense, thanks for clarifying

The mixture of experts approach was found to be inferior than a straightforward transformer. However, researchers discovered that MoE models give substantially better results when guided by fine-tuning. My guess is that GPT-4 was an experiment to prove out that theory at huge scale and I also suspect that its performance astonished OpenAI as much as everyone else.

If it was an experiment I'd like to know why I'm still paying for access to something claiming to be it.

Because you are getting value from it. Why else would you pay?

Because you signed up to pay for a beta service???? You can stop paying at any time.

If you're paying for it, it's a released product, not a beta. No matter how much a company wants to claim it's a "beta", it's simply not. The company just wants to be able to adhere to lower standards.

OpenAI is hardly the only company that pulls these shenanigans, too.

No, beta has nothing to do with if it's paid or not. Lots of products are released, and you pay for them, with early access/ beta.

No one lied to you. No one tricked you. It is in beta. You chose to pay for it.

I never said anyone lied to or tricked anyone. I'm saying that "beta" is a term referring to pre-release testing. If people are paying for it, it's been released and is therefore no longer a beta.

Calling it a "beta" at that point is just pure PR.

> If people are paying for it, it's been released and is therefore no longer a beta.

You can define "beta" whatever weird way you want but don't complain when it's not how literally everyone else uses it and don't complain when you're paying for something that uses the term the way everyone else does

My "weird way" is old-school, I admit, but it's not weird. It used to be mainstream. Google led the charge to deprive "beta" of actual meaning. I won't stop pushing back on it, because the redefinition makes it more difficult to talk about technical issues.

> don't complain when you're paying for something that uses the term the way everyone else does

I won't, because I won't pay to be a beta tester in the first place. I don't care what others do.

> because I won't pay to be a beta tester in the first place.

Apparently you will, because you started this thread with "why am I paying for this beta software?".

this is just not at all true -- companies have private betas that they monetize all the time. yes, the company provides lower guarantees on the product being supported. but the customer gets to use a product far before they normally would if they waited for general availability. that's the core of the idea behind a beta, not related to if it's paid or not

As I said, doing this is not rare and is certainly not limited to OpenAI. But it's misusing the term "beta" pretty significantly.

It's actually 16 GPT 3.5s in a trenchcoat where each one is slightly different, like the minions.

Can anyone elaborate on the specialization? Does it have to do with segmenting the dataset, or with training on different kinds of tasks?

Fine tuning to a specific more constrained set of tasks is my understanding.

Reply on news.ycombinator.com