Comment by visarga | Hacker News Reader

Comment by visarga | Hacker News Reader

Parent Full thread

> AI labs are constantly looking for new sources of high quality data

OpenAI has 200M users, and solve over 1B tasks per month interactively. This amounts to 1-2 trillion mixed human/AI tokens. The fact is that every user has their own unique life experience, and a reservoir of tacit knowledge they didn't communicate or write down anywhere else. The LLM can elicit that tacit knowledge that would be otherwise lost, it can crawl our minds for ideas and problem solving choices.

LLMs are in a situation of indirect agency. When they propose a solution, the human usually takes it out and implements it in the real world, and comes back for more help communicating the outcomes. Across many sessions it becomes possible to check what AI ideas worked out and what ideas were bad. This is a huge resource, it collects experience from every user. The LLM becomes an experience flywheel, people are attracted to the best models, and they will get the lion share of this experience.

And yes, you can do it with privacy in mind. You can train just a preference model instead of supervised training on chat logs. Just a model that would pick the right answer from a lineup. This way PII and user specifics don't leak.

Such a service is the opposite of a flywheel (a brake?) in practice. Those tokens are extremely low quality data.

1) I strongly double if even one in a thousand people actually bothers to report the outcome of their result. I and every programmer I know certainly don't (if we give up or it works we both just stop asking). And I'd think its one in a hundred thousand who gives detailed feedback that is useful for training. And one in ten million an expert that sits down and talks about its deep knowledge with it (and the problem of how you deduce this from chats that it isn't some crazy person).

2) AI's are extremely confident in their answer, which fools many people, especially those in a void of knowledge. Even if people did tell OpenAI whether every solution worked or not, I would heavily discount the accuracy of such data.

3) AI output autocannibalism does not lead to better outcomes, AI companies avoid using AI data for training like the plague. Mixed tokens I doubt would be much better.

The situation in reality is something like one in some huge number - maybe hundreds of thousands - of mixed tokens can be useful. Of those, those that are repeats of high quality sources like textbooks, dictionaries, man pages - have no value. Of the remaining, there is huge problem of how do you extract these needles in the haystack with high confidence. Given the incredibly lopsided confusion matrix (with the massive amount of actual negatives to actual positives) and this incredibly unstructured data set, I doubt its even remotely possible to find a way where you don't end up with a totally unacceptable ratio of actual to false positives. Letting this kind of garbage data in is how you get Gemini's gasoline spaghetti.

“AI companies avoid using AI data for training like the plague”

That’s not accurate. All of the big LLM training labs are leaning increasingly into deliberately AI-created training data these days. I’m confident that’s part of the story behind the big improvements for tasks like coding in models such as Claude 3.5 Sonnet.

The idea of “model collapse” from recursively training on AI-created data only occurs in lab conditions that very deliberately set up those conditions, from what I’ve seen. It doesn’t seem to be a major concern in real-world model training.

> Such a service is the opposite of a flywheel (a brake?) in practice. Those tokens are extremely low quality data.

I think not. Say you ask the model to help solve a coding problem. It gives you an idea, you try, it fails, come back and iterate. They can save a note for later finetuning - what worked and what didn't work, using you the user as a validation system for the LLM.

But you might also have your own experience and help the model where it struggles, and finally achieve the task. That is how the model can borrow both your experience and your manual validation work to improve itself.

Some tasks are spread over multiple sessions, or multiple days. They can cluster and look at your progress over time. The latter steps provide rich feedback on the quality of the former steps. Hindsight is 20/20.

Even in chats where the user doesn't perform validation there is rich feedback, people share some of their tacit experience. It's a form of delayed feedback, humans act as caches of unique experience.

The way I conceptualize this is as a search process - problem space search. LLMs can search better with assistance, and humans also search better with assistance. LLMs collect experience from millions of people, they funnel experience into their logs.

Reply on news.ycombinator.com