Comment by MuffinFlavored | Hacker News Reader

Comment by MuffinFlavored | Hacker News Reader

Parent Full thread

> What is OpenAI's moat?

From what I understand, if you take the absolute best cutting edge LLM with the most parameters and the most up to date model from GitHub/HuggingFace/whatever, it's very far off from the output you get from GPT-3.5 / GPT-4

aka full of hallucinations, not very useful

I don't know if this is the right way to look at it but if what George Hotz said about GPT-4 simply being "8 220B parameter models glued together by something called a mixture-of-experts", from what I understand, OpenAI's moat is:

their access/subsidiized cost to GPUs/infrastructure with Microsoft

the 8 220B models they have are really good/I don't think anything open source matches them/nobody can download "all of Reddit/Twitter/Wikipedia/StackOverflow/whatever else they trained on" anymore like they could given how everybody wants to protect/monetize their content now

and then the "router" / "MoE" piece seems to be something missing from open source offerings as well

What if specialized smaller models is the best way ahead for community. I don't care if I am interacting with one big model which can do everything or I have to go to different websites to access specific models. All model sizes will be useful. Smaller models will be frequently used. Bigger less so.

This isn’t really true (or at least, doesn’t apply across the board). Qwen (Alibaba’s open source model) outperforms GPT4 on Chinese language tasks, and I can further finetune it for my own tasks (which I’ve done, and I confirm it’s produces more natural output than GPT4).

Other benchmarks/anecdotes suggest fine-tuned code models are outperforming GPT4 too. The trend seems to be that smaller, fine-tuned task specific models outperform larger generalised models. It requires a lot of resources to pretrain the base model, but as we’ve seen, there’s no shortage of companies who are willing and able to do that.

Not to mention, all those other companies are already profitable, whereas OpenAI is already burning investor cash.

Depending on the task, the best open models will outperform GPT-3.5, but would be more expensive to run at comparable speed. GPT-4 is in a league of its own.

Reply on news.ycombinator.com