Hacker News Reader

Comment by paxys | Hacker News Reader

What is the bottleneck in building the "largest LLM" today for any interested party? AI expertise? Training data? GPUs?

GPUs and electricity. GPT-4 reportedly cost 100M$ to train.

I'd say GPU and people trained in knowing what to do while training. There is a lot of money in the world, as we see Microsoft and other companies handing it out. Power even, if you build in the right place can be cheap, see bitcoin mining.

Very much agree... I think people underestimate how finicky these models are to train. So much so that one of the big announcement around GPT-4 was the fact that OpenAI found a way to make it smoother and more predictable.

As evidence by this 114 page log of how engineers resolved problems that came up during the training of OPT-175, which lasted for several months. https://github.com/facebookresearch/metaseq/blob/main/projec...

It has to be more than just that. Bard is clearly inferior to GPT-4 and Google has no shortage of GPUs or electricity (or data).

> Bard is clearly inferior to GPT-4

Have you tried it recently? It got a lot better, and I like it more for research tasks. (GPT4 + Bing are alright)

It’s actually insanely good now, especially with images

$100m is relatively cheap for something at the scale and sophistication of gpt-4. I think money is not the issue-it’s all the little things that had to be done right as Altman said in an interview.

So one can burn $100m and not have gpt4. Google.

They're working on Gemini which is supposed to blow gpt-4 out of the water. I believe it when I prompt it.

We have seen this happen with Google+. Only Google pawns trying to convince people to use it.

H’s for the RLHF too?

I guess that is easy to solve by throwing money

I believe it's it's humans (for RLHF and other things) and data. If it were just power and chips, the current front-runner would be a corporate behemoth or a state, not OpenAI. (They might become a corporate behemoth soon enough but they aren't so far.)

At this point I'm pretty convinced that no new gigacorp will be emerge out of this generation of AI (LLM and vision models). OpenAI will have to continuously bring their A-Game to stay 6 months ahead of everyone else. There is durable moat that they could hide behind in sight as of now. Of course they would like to construct a regulatory moat but it doesn't seem like there is a clear majority for this.

I'm pretty sure and happy that this will probably turn out as a large consumer windfall. Actually comes right in time to counteract the strong inflationary forces that come with de-globalization and declining efficiency due to increasingly basing the economy on gov spending.

I'd say expertise and compute. With enough compute I can certainly build the largest LLM. Do I have the expertise to build the BEST largest LLM? No.

Easy. Human feedback and training. The open secret is that just the AI “revolution” is built upon uncompensated or barely-compensated human labor. Not just everyone uploading their stuff to the internet for 30 years but also this:

https://www.forbes.com/sites/kenrickcai/2023/04/11/how-alexa...

Most people on HN have no idea about that

https://futurism.com/the-byte/ai-magic-spy-on-you-experts-wa...

And especially OpenAI:

https://time.com/6247678/openai-chatgpt-kenya-workers/

That’s the dirty secret of why ChatGPT 4 is better. But they’ll tell you it has to do with chaining ChatGPT 3’s together, more fine tuning etc.

They go to these poor countries and recruit people to work on training the AI, scan their eyeballs for crypto and whatever else. And then privatize the profits from all this.

But it’s nothing new, capitalists have always exploited labor… and people make their choices.

At least it isn’t ivory or blood diamonds

https://www.reuters.com/technology/kenya-panel-urges-shutdow...

Money?

Plenty of people have money, but they don't all have world class LLMs. Clearly money isn't the only problem.

Not everyone who has money wants to build one .

It is just a shiny new toy, they may deem there are better uses for their money

Reply on news.ycombinator.com