Hacker News Reader

>We are basically out of new, non-synthetic text to train models

this is not even remotely true.

There is an astronomical amount of data siloed by publishers, professional journals etc. that is yet to be tapped.

OpenAI is making inroads by making deals with these content owners for access to all that juicy data.

Even assuming there is a ton of data companies are just now getting access to, the logarithmic curve of LLM improvements is clearly visible (granted that our LLM evaluation frameworks are not very good)

>>>There is an astronomical amount of data siloed by publishers, professional journals etc. that is yet to be tapped.

You seem to think these models haven't already been trained on pirated versions of this content, for some reason.

Yep, books3 is what llama was famously trained on before it was taken down.

That’s not even considering AI crawlers or all the copyright text on archive.org

Reply on news.ycombinator.com