Hacker News Reader

Comment by tunesmith | Hacker News Reader

I've been curious about this for a while, I have a hobby use-case of wanting to input in-progress novellas and then asking it questions about plot holes, open plot threads, and if new chapter "x" presents any serious plot contradiction problems. I haven't tried exploring that with a vectordb-embeddings approach yet.

View on HN

This is an exact example of something a vector dbs would be terrible at.

Vector dbs work by fetching segments that are similar in topics to the question, so like "Where did <Character> go after <thing>" will retrieve segments with locations & the character & maybe talking about <thing> as a recent event.

Your question has no similarity with the segments required in any way; & it's not the segments that are wrong it's the way they relate to the rest of the story

Good points - LLMs are ok at finding things that exist, but they have zero ability to abstract and find what is missing (actually, probably negative; they'd likely hallucinate and fill in the gaps).

Which makes me wonder if the opposite, but more laborious approach might work - request it identify all characters and plot themes, then request summaries of each. You'd have to review the summaries for holes. Lotsa work, but still maybe quicker than re-reading everything yourself?

Firstly, I don't at all agree that they have zero ability to abstract. Doesn't fit my experience at all. A lot of the tasks I use ChatGPT for is exactly to analyse gaps in specifications etc. And have it tell me what is missing, suggest additions or ask for clarifications. It does that just fine.

But I've started experimenting with the second part, of sorts, not to find plot holes but to have it create character sheets for my series of novels for my own reference.

Basically have it maintain a sheet and feed it chunks of one or more chapters and asking it to output an a new sheet augmented with the new details.

With a 100K context window I might just test doing it over while novels or much larger chunks of one.

> LLMs are ok at finding things that exist, but they have zero ability to abstract and find what is missing (actually, probably negative; they'd likely hallucinate and fill in the gaps).

I feel this is mostly a prompting issue. Specifically GPT-4 shows surprising ability to abstract to some degree and work with high-level concepts, but it seems that, quite often, you need to guide it towards the right "mode" of thinking.

It's like dealing with a 4 year old kid. They may be perfectly able to do something you ask them, but will keep doing something else, until you give them specific hints, several times, in different ways.

There are other ways of using the model other than iterative forward inference (completions). You could run the model over your novel (perhaps including a preface) and look at the posterior distribution as it scans. This may not be so meaningful at the level of the token distribution, but there may be interesting ways of “spell checking” at a semantic level level. Think a thesaurus but operating at the level of whole paragraphs.

that's not at all what I said

Do the OpenAI APIs support converting prompts to vectors, or are people running their own models locally to do this? Can you recommend any good resources to read up on vector DB approaches to working around context length limits ?

Indeed this tutorial on Haystack is a good one as an example: https://haystack.deepset.ai/tutorials/22_pipeline_with_promp... It combines a retrieval step followed by a prompt layer which inserts the relevant context into the prompt. You can however change the 'retrieval step' with something that uses a proper embedding model and OpenAI also provides those if you want to. I tend to use lighter (cheaper) OSS models for this step though. PS: There's some functionality in the PromptNode to make sure you don't exceed prompt limit.

That's great - thanks!

Yes, you can use a local embeddings model like gtr-t5-xl alongside retriever augmentation. This can point you in the right direction: https://haystack.deepset.ai/tutorials/22_pipeline_with_promp...

Thanks!

open ai has an embeddings api that ppl use for that https://platform.openai.com/docs/guides/embeddings, though whether it's the best model to do that is congested.

Contriever is an example of a strong model to do that yourself. see their paper too to learn about the domain. https://github.com/facebookresearch/contriever

Thanks!

So what's the right way to do a wider-ranging analysis? Chunk into segments, ask about each one, then do a second pass to review all answers together?

Reply on news.ycombinator.com