Hacker News Reader

Comment by crazygringo | Hacker News Reader

Can someone here explain how an existing model like ChatGPT, that you only have API access to, can be used to train your own model and somehow copy aspects of it?

That's entirely non-intuitive to me how that would work. Like are they just asking it questions about every topic under the sun, and then creating training materials out of the answers? How would you even begin to assemble a list of prompts that would cover all of knowledge? And how could you ever distinguish useful outputs from nonsense hallucinations?

I feel like I'm missing some key details here that the article and its links don't explain at all.

View on HN

Not an expert but I suspect it would be most useful for the instruction tuning and fine-tuning phases. They would pre-train the model on some huge dump of stuff from the internet like the Pile -- Wikipedia, books, Reddit scrapes and so on, and the result would be something that can predict next words really well but doesn't know it's meant to answer questions. That's the pre-train. The next step is to train that on transcripts of chats so that it specialises in those, and finally on chats that have the balance of helpfulness and harmlessness that you want. Those latter two need significantly less data, but still more than you can cheaply produce with humans, so they could use OpenAI to generate it. But perhaps it's a small enough amount that even if you need an LLM to generate it economically, it's affordable to have humans read through it and throw away anything excessively hallucinated. Or perhaps you can restrict the scope of the conversations to ones where hallucinations are unlikely, and trust that fine tuning on them won't prevent the LLM from digging into its deeper knowledge in use.

Bytedance will still have access to the same open internet as OpenAI had. But it does not have the weights and the parameters.

However by asking ChatGPT questions and recording their answers, you can use that to fine-tune the model they are creating. They can tell their own model to answer the way ChatGPT answered.

In short they are copying how ChatGPT would answer. and not finding out for themselves by creating their own Question and answer (or completion) datasets.

That is what RLHF is. But anyway, Chinese companies will be able to compete here, as this will be commoditized quickly and China is the king of creating and selling commodities.

> How would you even begin to assemble a list of prompts that would cover all of knowledge?

They don't need to, all that knowledge is already in public training sets from scraping the internet.

What is harder to get is the answer patterns to ensure good user experience. You want a lot of such answer patterns so the model knows what structure to use for different kinds of questions. That structure isn't to make the result formatted for humans, but contains reasoning paths the LLM takes in order to arrive at reasonable answers. Since an LLMs thinking is the words it writes the structure of how it responds thus corresponds to thinking patterns, and you want the LLM to learn a lot of those, internet data wont contain that but ChatGPT responses will.

TLDR: The words an LLM writes is also what it thinks, since it doesn't hide thoughts, so by training on what ChatGPT writes you also train on how it thinks. It isn't the facts you want but the thinking patterns.

Reply on news.ycombinator.com