Hacker News Reader

Hacker News

Comment by checkyoursudo | Hacker News Reader

Parent Full thread

checkyoursudo

Was it ever confirmed whether GPT-4 is a Mixture of Experts or not?

View on HN

I just asked GPT-4, and it denies being a MoE model.

User:

Are you an MoE model?

ChatGPT:

No, I am not based on a Mixture of Experts (MoE) model. My underlying architecture is based on the GPT (Generative Pre-trained Transformer) framework, specifically the GPT-4 version. This architecture is a large-scale transformer-based neural network, but it does not use the MoE approach.

In a GPT model like mine, the entire model works in a unified manner on each task, without dividing it among specialized sub-models or experts. The model is trained on a diverse range of data and tasks, enabling it to generate responses across a wide array of topics and questions. While this approach lacks the specialization that MoE models offer, it provides broad, general-purpose language understanding and generation capabilities.

LLMs don’t have insight, these outputs cannot be assumed to be accurate.

Due to current hardware limitations it is not feasible to have a 1T parameter model without MoE.

I wonder if it has some canned human-written responses when asking specific questions about itself. This would be pretty clever to silently implement, that will definitely help convince people that it's approaching "AGI".

It's possible that it's just hallucinating here too, I don't have any proof that the responses are canned, but they appear that way to me.

Yes, it has an invisible "shadow prompt" that gets sent to it when you start a session. It looks something like this:

https://www.reddit.com/r/ChatGPT/comments/zo9of4/comment/j0n...

That makes sense, there has to be some level of human written responses for specific things, like when you ask it about a controversial topic that it refuses to answer, it gives you some canned response about how it won't answer that, and that very obviously isn't what language models do naturally. For these cases there has to be some sort of pre-programmed human generated reply!

It's interesting that other people don't seem to agree with the idea that it has some pre-programmed responses, I am curious as to what they think is going on here.

That is almost surely a hallucination.

Curious, why do people gravitate to the term "hallucination" (as if it's a person not a tool) instead of just plainly stating that it's wrong?

If I ask it what color an orange it and it says blue, that would be wrong.

If you ask it a question and it makes up a completely fabricated story, like for example the case files in that recent legal case [1], then saying it was “wrong” doesn’t really seem to capture it.

Calling it a hallucination is a great analogy, because the model made up a plausible sounding, but completely fabricated story. It saw things that were never real.

[1] https://apnews.com/article/artificial-intelligence-chatgpt-c...

Fair enough and thanks. I felt hallucination was too forgiving a term but I can see how others would rank them the other way around and suppose it works.

The more accurate term I’ve seen floating around is “confabulation”. It’s also a human phenomenon, but it doesn’t bring all the baggage of hallucinations.

It’s a chat bot so a lot of language will be anthropomorphic. Even still this definition fits well “3. a false belief or impression; illusion; delusion.”

It does not and it cannot answer this kind of meta-question unless explicitly added to its prompt/RAG or whatever.

And OpenAI would not add such proprietary information there.

Ask it to produce a citation for that claim so we can verify it.

Can you explain how your own mind works?

Reply on news.ycombinator.com