Hacker News Reader

Comment by ldjkfkdsjnv | Hacker News Reader

OpenAI has a big problem, the only moat on an LLM is the quality and inference speed. Which is quickly getting commoditized.

View on HN

The comment above seems to be interpreting OpenAI’s goals in a way that is not consistent with its charter.

Structurally, for OpenAI, capped profit is a means not the end. If the capped profit part of OpenAI is not acting in alignment with its mission, it is the OpenAI board’s responsibility to rein it in.

—

https://openai.com/charter

OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity. We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome. To that end, we commit to the following principles:

Broadly distributed benefits

…

Long-term safety

…

Technical leadership

…

Cooperative orientation

Turboderp released a custom quantized version of dbrx-base and instruct a few days after the official model landed. https://huggingface.co/turboderp/dbrx-base-exl2

You are right, LLMs weights are fast getting commoditized. As fast as compute is scaling now, I don't think it can continue forever, so the huge advantages big players have is going to get wiped. I'm sure they will always be able to deliver at scale, but a super GPT4 performing model available on your local (or small player type) hardware in the next few years seems more likely than a few big mega-players and mega-models.

Is it? I thought this would be the case by now but GPT-4 is still in the lead (and released a while ago). How many generations in a row does OpenAI have to dominate before we revisit the "there is no moat" idea?

i've been a paid consumer of chatgpt - 4 for a while and i've been recently using gemini more and more, its free and while not as good as gpt-4, it is good enough where paying for gpt-4 doesn't seem worth it anymore.

Anthropic Opus is better than GPT 4 on many (most?) tasks.

Not only better as measured by benchmarks, but also better preferred by humans.

https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

The most surprising thing to me is that Opus is only slightly in the lead.

I was feeding multiple python and c# coding challenges / questions to both and Opus blew GPT4 out of the water on every single task. Didn’t matter if I was giving them 50 lines or 5,000 Opus would consistently give working/correct solutions while GPT4 preferred to answer with pseudo code, half complete code with ‘do the thing here’ comments, or would just tell me that it’s too complicated.

Another data point, I definitely find Opus better for coding, but not by much. The problems I give them are generally short (<= 100 lines) and well-defined so any advantage Opus has in larger contexts won't be apparent to me. They're also generally novel problems but NOT particularly challenging (anyone with a BS CS should be able to solve them in < 1hr).

I have them working with mostly C++ and Clojure, a bit of Python, and Vimscript every once in a while. Both models are much better at Python and fairly bad at Vimscript. Clojure failure cases are mostly from invented functions and being bad at modifying existing code. I can't pick out a strong pattern in how they fail with C++, but there have been a few times where GPT4 ends up looping between the same couple unworkable solutions (maybe this indicates a poor understanding of prior context?).

Spot on. People need to say what they are actually using the models for and not just "coding".

I mostly use it to make react/javascript front ends to a python/fastapi backend and chatGPT4 is great at that.

I tried to write a piece of music though in the old Csound programming language and it barely even works.

It will be interesting to see how the context plays out because I have noticed that I can often give it extra context that I think will be helpful but end up causing it to go down a wrong path. I might even say my best results have been from the most precise instructions inside the smallest possible context.

It's because LMSYS is an aggregate elo across a range of different tasks. Individually in some very important areas, Claude Opus may be better than GPT-4 by 50-100 elo points which is quite a lot. However there are specific domains where GPT-4 has the advantage because it's been fine tuned based off a lot of existing usage. So weak points around logic puzzles or specific instructions don't bring down its elo whereas Claude Opus doesn't have this advantage yet. I believe Opus's eventual elo, after all these little areas of weakness are fine tuned, will be something like 1300.

Yeah GPT is incredibly lazy, ironically 3 is far better at not being lazy than 4.

I guess you benchmarked via API? I've heard even the datestamped models have been nerfed from time to time..

Same here. I pay for both GPT-4 and Opus. Opus is miles ahead on software development work for me.

Agreed. Opus definitely has taken most of my usage as of late.

We don't talk about Google "not having a moat" since their patents on PageRank expired.

All the talk of OpenAI's moats (or lack thereof) since the memo, seems like humans being stochastic parrots.

Google has a data moat on traditional search. Its really hard to train a traditional search system without google scale data. LLMs effectively side stepped that whole issue

Google's special sauce, is the data of all user's responses to the things Google's algorithm shows them.

This description also works for ChatGPT, what it has which others do not (at anything close to the same scale).

Google's search engine is a well understood bit of matrix multiplication on a scale many others can easily afford to replicate.

This description also works for GPT-3.

No becuause ChatGPT was trained on the whole internet using a new algorithm. You needed perfect tagged data for the old algorithm (page rank). LLMs are a fundamental leap forward

The GPT algorithm isn't a secret.

The RLFH data — users giving a thumbs up or down, or regenerating a response, or even getting detectably angry in their subsequent messages — is.

PageRank isn't a secret.

Google's version of RLFH — which link does a user click on, do they ignore the results without clicking any, do they write several related queries before clicking a link, do they return to the search results soon after clicking a link — are also secret.

That the Transformer model is a breakthrough doesn't make it a moat; that the Transformer model is public doesn't mean people using it don't have a moat.

Hence why I'm criticising the use of "moat" as mere parroting.

Google's acquisitions are its moat. Google Search is only one small part of that

Google wouldn't be the company it is if it didn't also control Youtube, Gmail, Android and Chrome.

Yep, Groq is one example

Great example: can't scale enough to remove their 30 requests per minute (!) rate limit and enable billing, barely meets 3.5 levels of intelligence, etc;

People don't seem to understand that scaling out LLMs efficiently is it's own art that OpenAI is probably learning lessons on faster than anyone else

You're thinking of Grok. Not Groq. Blame Elon for this naming catastrophe.

Lol what? No, I'm thinking of Groq.

I love Groq though, every time someone hypes up their project using it you instantly know they have no real usage whatsoever.

Even the most "toyish" toy project doesn't fit in their rate limits for anything more than personal use.

Reply on news.ycombinator.com