Isn’t part of the point of GPT that it finds relationships without the training data having to be well-structured? So long as the text describes them in a way that a human, having read the books, could answer the question?
I don’t really understand how training works. This isn’t a jab.
Right now, I don't train anything, I've broken the text down to n characters and created embeddings for that subset of text -- then I search for the closest distance / relationships between the question asked. Then I add the text to the prompt, and tell gpt to use those paragraphs to answer the question (to ensure that it doesn't make anything up). This is one of the ways I can get around the token limit, but it comes at the cost of thinking it can only use the paragraphs I show it. I'm trying to improve the prompt to get more consistent results, and maybe 4 can help me give it larger bodies of text!
Hope that answers your question, let me know if you have any more!
So your software takes a prompt from me, does non-GPT work to find additional context from your source (the books, parsed and re-structured into word vectors or whatnot), and then asks GPT my prompt combined with the added context?
Like,
“What are the three foobars when considering these passages from a book <…> ?”
- Your question -> vectors with open ai embeddings - Text you uploaded before -> vectors with open ai embeddings
Get the most similar above a certain threshold, and then add it to the prompt saying
"From these articles / paragraphs, answer the user's question: What are the three foobars"
So yep! I preprocess it
It feels like training is analogous to years of growing up and going school. And what you’ve done is taken that educated “mind” and said, “here’s a document. I’m going to quiz you on it.”
That seems really practical compared to sending an AI back to school to learn all about some specific topic.
I wonder if we will end up with a library of trained models, each of which went to different schools to obtain different focuses of knowledge. Maybe LiteratureGPT is better suited for this than General GPT or ProgrammerGPT.
Okay I think I’ve stretched the analogy far enough.