Hacker News Reader

Comment by loveparade | Hacker News Reader

What's most surprising to me is that OpenAI really seems to believe that not publishing details will save them from competition. Everyone knows how these models work, and while I'm sure there is a bunch of "secret sauce" that OpenAI has built for training and fine-tuning, it's ridiculous to believe that the research community and competitors like Google and Facebook can't figure out the same. They just haven't really tried until recently because the capabilities and ROI of these models weren't obvious. No matter who you are, most of the smartest people work for someone else.

The only competitive advantage that OpenAI has here is a headstart of 6-12 months from all the infrastructure investment into training these kinds of models. Now that everyone wants to build competing models with the same capabilities, this advantage is going to disappear very quickly.

View on HN

Au contraire, no one knows how large GPT-4 is, which is the single best predictor of performance (for a model trained to convergence). The GPT-4 paper spent much of its time writing about this — they did some small scale experiments with 1/1000th the compute, then picked a loss level they wanted and trained GPT-4 till it got it.

Neither the exact loss level nor the number of parameters are revealed by the paper. Unfortunately it’s not possible to guess these from outside observations.

Will this save them from competition? No, but it certainly makes things harder. Everyone immediately aimed at 175B the moment GPT-3 was published. GPT-4 is now a question mark.

I can't believe anyone considers a single number, which would work about equally well if it were 10% higher or lower, to be a trade secret.

This is not really true. The Chinchilla paper showed that a 4% difference in loss between Chinchilla and Gopher led Chinchilla to blow Gopher out of the water at most tasks, including 30x performance in physics.

Empirically, LLMs have shown to have emergent abilities appear at different loss levels. So, a 10% difference could really matter.

That ten percent is not loss it is parameter count.

It's about causing your competition to waste millions of dollars in compute time and power doing something unproductive.

There is not a huge pile of excess TPUs laying around for people to use. Any strategic advantage can quickly compound and put you well ahead of others.

For all we know they have hit 500B parameters with some clever unpublished optimisation, which would both give them an edge and if revealed would put a damper on the preveiling belief that LLMs can scale and scale (eg. 3x more params for less than 3x performance).

As you say, there is absolutely no way for us to find out.

I think the big tech actors probably know. Information leaks and ultimately Google is spyware. Not that it will reach the public knowledge today, but that kind of information is difficult to keep in the bottle long time.

Then what's everyone so mad about?

What about the training data corpus ? Other than large cos like Google or Meta, can anyone else procure the same ?

Leaving the legal aspects of crawling aside, I think there is an important distinction here between 1. "can you procure it" and 2. "do you have enough money to process it all"

1. Yes, I think almost anyone can write code to procure the training corpus, in theory, and test it on a small scale

2. No, only the biggest labs and universities have enough resources to process such huge amounts of data and iterate on models with that scale. But that's just a matter of resources that can be overcome with partnerships between industry and academia that are common anyway. All the big labs already have huge efforts underway to reproduce GPT-X and it's just a matter of time before they catch up.

At least as far as what the GPT-3 papers claimed, all (or most?) of the data used for training would be freely available for other competitors/researchers to acquire. Wikipedia, Common Crawl data, etc. I don’t believe OpenAI did their own crawling at all.

With OpenAI not being really open, it’s hard to say for sure what exactly ended up in the training materials, though. GPT-4 is even more of a black box to anyone outside of OpenAI with very little information released on how it was trained.

You say: OpenAI really seems to believe that not publishing details will save them from competition.

Then say: The only competitive advantage that OpenAI has here is a headstart of 6-12 months

It's almost as if they want to keep this advantage, huh? Blows my mind how business illiterate some HN commenters are.

If someone doesn't file a Form 990, they are out for themselves and want to fuck (sorry... extract value from) everyone who isn't a (majority) share holder.

Open AI is not the first for-profit philanthropy. It's just another evangelical church with a televangelist at the helm. TED talks are sermons.

TBF: at least Open AI doesn't pretend to be a charity anymore... I feel sorry for the working sops who held MSFT stock in 401Ks and funded a massive tax write-off for the capital class. Dumbasses, amirite?

A headstart doesn't matter unless you can keep it. The point is that there are many mort smart people and resources outside of OpenAI and there are inside of OpenAI. If they focus their efforts, they will easily catch up.

A headstart is not a competitive moat like network effects are. Go and try to raise money for your startup from a VC and tell them "well, everyone is doing the same as us, but we started 6 months earlier!!" - nobody cares.

So OpenAI should just give up and release all their trade secrets? To what purpose?

I dunno, maybe the Open part of OpenAI should hint at it.

The problem people are having is that OpenAI marketed themselves as supposedly democratizing AI, but it does the opposite.

It turns out most individuals will give up democracy if they think they have a shot at becoming king.

That, or rename themselves to ClosedAI.

Reply on news.ycombinator.com