Comment by visarga | Hacker News Reader

Comment by visarga | Hacker News Reader

Parent Full thread

I recently worked on information extraction from 10K documents. GPT-3 needs about 7 days of operation in batch mode on one thread. It takes 40..70s to read one single document and report the extracted data. One MINUTE per page.

But I think you meant GPT-3 has seen many books during training, not during inference. You should know that training on millions of books is not the only way GPT-3 learns. It is just the foundation of its knowledge.

GPT-3 learns "in-context", that means it can learn a new word or a new task at first sight. It just needs a description or a few examples. This is the most powerful feature of GPT-3 - in-context learning. And when it comes to ICL, it is much like humans - only sees a few examples, not millions of books.

> “Do you ask for permission when you train your mind on copyrighted books?”

The nature of ICL is that it happens at prediction time. So GPT-3 would have to explicitly be instructed to learn a specific skill. Should it reject instructions if they are sourced from copyrighted books?

> “You should know that training on millions of books is not the only way GPT-3 learns. It is just the foundation of its knowledge.”

I’m not a lawyer, but to me it seems within the realm of possibility that a U.S. court eventually finds strongly in favor of the copyright holders, the Supreme Court agrees (because Big Tech has so few friends left), and OpenAI will be required to destroy the GPT-3 model and all copies of the training data because they can’t filter out copyrighted works.

Yeah, I think "we can't actually tell which bits of our model are derived from your work which we copied without your permission onto our system for training purposes probably makes AI companies more vulnerable rather than less. Other platforms like search and social media have successfully defended themselves by showing willingness to promptly remove copyrighted material, let copyright holders opt out of indexing or even negotiating stuff like ContentID so copyright holders get paid each time their work is used.

Your reply is mostly a red herring because as you yourself said, the OP was talking about regular training data, not the ICL you focused on.

Just because you can find one way that GPT might be slow doesn't invalidate the point that its training does use massive amounts of data.

If GPT-3 can learn in context it means both the training set and the prompt could be in copyright violation. So even a clean model, trained on licensed data, cannot guarantee there will be no copyright issue.

So what if it does?

Reply on news.ycombinator.com