Hacker News Reader

Hacker News

Comment by pulse7 | Hacker News Reader

Full thread

pulse7

Isn't GPT-4 already over 1T parameters? And GPT-5 should be even "an order of magnitude" bigger than GPT-4...

View on HN

Was it ever confirmed whether GPT-4 is a Mixture of Experts or not?

I just asked GPT-4, and it denies being a MoE model.

User:

Are you an MoE model?

ChatGPT:

No, I am not based on a Mixture of Experts (MoE) model. My underlying architecture is based on the GPT (Generative Pre-trained Transformer) framework, specifically the GPT-4 version. This architecture is a large-scale transformer-based neural network, but it does not use the MoE approach.

In a GPT model like mine, the entire model works in a unified manner on each task, without dividing it among specialized sub-models or experts. The model is trained on a diverse range of data and tasks, enabling it to generate responses across a wide array of topics and questions. While this approach lacks the specialization that MoE models offer, it provides broad, general-purpose language understanding and generation capabilities.

That is almost surely a hallucination.

Curious, why do people gravitate to the term "hallucination" (as if it's a person not a tool) instead of just plainly stating that it's wrong?

It’s a chat bot so a lot of language will be anthropomorphic. Even still this definition fits well “3. a false belief or impression; illusion; delusion.”

If I ask it what color an orange it and it says blue, that would be wrong.

If you ask it a question and it makes up a completely fabricated story, like for example the case files in that recent legal case [1], then saying it was “wrong” doesn’t really seem to capture it.

Calling it a hallucination is a great analogy, because the model made up a plausible sounding, but completely fabricated story. It saw things that were never real.

[1] https://apnews.com/article/artificial-intelligence-chatgpt-c...

Fair enough and thanks. I felt hallucination was too forgiving a term but I can see how others would rank them the other way around and suppose it works.

The more accurate term I’ve seen floating around is “confabulation”. It’s also a human phenomenon, but it doesn’t bring all the baggage of hallucinations.

Ask it to produce a citation for that claim so we can verify it.

LLMs don’t have insight, these outputs cannot be assumed to be accurate.

Due to current hardware limitations it is not feasible to have a 1T parameter model without MoE.

I wonder if it has some canned human-written responses when asking specific questions about itself. This would be pretty clever to silently implement, that will definitely help convince people that it's approaching "AGI".

It's possible that it's just hallucinating here too, I don't have any proof that the responses are canned, but they appear that way to me.

Yes, it has an invisible "shadow prompt" that gets sent to it when you start a session. It looks something like this:

https://www.reddit.com/r/ChatGPT/comments/zo9of4/comment/j0n...

That makes sense, there has to be some level of human written responses for specific things, like when you ask it about a controversial topic that it refuses to answer, it gives you some canned response about how it won't answer that, and that very obviously isn't what language models do naturally. For these cases there has to be some sort of pre-programmed human generated reply!

It's interesting that other people don't seem to agree with the idea that it has some pre-programmed responses, I am curious as to what they think is going on here.

It does not and it cannot answer this kind of meta-question unless explicitly added to its prompt/RAG or whatever.

And OpenAI would not add such proprietary information there.

Can you explain how your own mind works?

Yes, the article even mentions that. Guess they went with that headline to attract readers.

Was thinking the same, so 1T might bring you to the league of GPT-4. Actually in the best case, since it seems that meta, google, openai and so on have the most talent.

Anyway, to bring it to the next level, how big should it be? Maybe 10T? 100T?

> Maybe 10T? 100T?

I don't think we have enough training data to train models so big in a way to efficiently use all the params. We would need to generate training data, but then I don't know how effective it would be.

Have seen an interview with someone from openai. At least they claimed, that they are far away from running out of data and they are able to generate data if needed.

> they are able to generate data if needed.

If they can generate data they should use that instead of piping it through a million-dollar lossy compressor.

The point of generating data is so you can teach the model its distribution, not so it can memorize it.

Reply on news.ycombinator.com