ChatPDF – Chat with Any PDF
chatpdf.com
chatpdf.com
Wrote a <50 lines version with LangChain to run on your terminal with any folder full of PDF documents - https://github.com/angad/dharamshala/blob/main/docs.py
return_source_documents is particularly helpful to get a sense of what is being sent in the prompt.
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=300)
As some suggested in other comments, the tool probably processes paragraphs one by one so such injection need to be more sophisticated... maybe ChatGPT will think of some.
It uses langchain and pinecone to create a semantic index over the PDF content and search it based on question asked to sends the relevant information to openAI GPT api using embeddings.
I know of a few approaches for that:
- Ignore the problem and let it hallucinate answers to anything that's not in the first 5-10 pages
- Attempt to recursively summarize the PDF at the start - so summarize e.g. pages 1-3, then 4-6 etc, then if the resulting summaries are still too long for the context window run a summary of those summaries. Use the summary in the context to help answer the user's questions.
- Implement a mechanism for finding the most likely subset of the PDF content to include in the prompt based on the user's question. You could use the LLM to extract likely search terms, then run a dumb search for those terms and include the surrounding text in the prompt - or you could calculate embeddings on the different sections of the document and do a semantic search against it to find the most appropriate sections, as I did in https://simonwillison.net/2023/Jan/13/semantic-search-answer...
Which approach did you use? Am I missing any options here?
> In the analyzing step, ChatPDF creates a semantic index over all paragraphs of the PDF. When answering a question, ChatPDF finds the most relevant parapgrahs from the PDF and uses the ChatGPT API from OpenAI to generate an answer.
Are you using OpenAI's embeddings to implement that?
For a simple example, a car manual where you want to change the brakes, it probably won't tell you in the brake section how to remove the wheels. You have to look at the wheel section. And in the wheel section it won't tell you about the nuts, you have to look in the spec sheets. And the spec sheet won't have the torque, you have to look in the chapter reference.
Often times they are not nice enough to point you to the relevant sections, you just have to stumble around the manual for a long time.
1. User enters question 2. Semantic search for relevant sections of input material 3. Prompt LLM if it needs any further context to answer the question 4. GOTO 2 5. Finish
- Mathis from ChatPDF
That is incredibly interesting. We really need an Internet-scale semantic search engine API to try out this and make interesting LLM-based tools. Hooking up LLMs to classic keyword search engines like Bing and Google often gives underwhelming results.
Docalysis also shows you the PDF side-by-side, has page numbers, and overall responses are of better quality according to users that have emailed comparisons to ChatPDF.
https://news.ycombinator.com/reply?id=35628748&goto=item%3Fi...
PDF: https://raw.githubusercontent.com/sellout/recursion-scheme-t...
First message:
Hullo there! Welcome tae this PDF file aboot coElgot algebra and mair. Here are some questions ye might hae:
- Whit is the difference atween a catamorphism and an anamorphism?
- How daes a zygomorphism uise a helper function?
- Can ye gie an example o a refold in action?
If you are interested in this, feel free to reach out to me and I can help you with setting this up.
Uses weaviate so that even the vectorstore can be self-hosted
I suspect we'll see data leaks through misguided trust in AI models at some point in the near future, and it'll end up being a mess to clean up.
PDF: Sure, here's a blackout poem inspired by "Gödel, Escher, Bach: An Eternal Golden Braid":
Golden strands of music and math, A puzzle of paradoxes and paths. Recursive structures, self-reference, A journey through the mind's expanse.
Bach's fugues and Escher's art, Gödel's theorems that tear apart Our notions of truth and certainty, And lead us to infinity.
An eternal braid of ideas and themes, That weave together like a dream. A musico-logical adventure awaits, For those who dare to contemplate.
Hahaha, not bad :)
ChatPNG - apply OCR to an image, extract text, feed it to GPT. ChatMP3 - apply speech-to-text to a recording, feed it to GPT. ChatGPS - hmm. not sure yet. something location-based obviously...
If any VC's are interested, I'm selling 10% stake in these projects for only $20k right now. /s
> It uses ChatGPT API to generate color name from color hex.
It does use ChatGPT, but yeah, I get the sentiment.
jannies should start deleting these ads. ever since gpt-4 dropped, this website has become unbearable.
If you're worried about astroturfing, email the mods and they'll take a look.
By doing this, I was able to reduce costs (credit usage) significantly, while still achieving high performance. Plus, the smaller embeddings were faster and available for free. I only needed to call the OpenAI API for response generation, and by optimizing the context and selecting the most relevant search results, I was able to achieve highly accurate responses. Here's the repo: https://github.com/mallahyari/drqa
A word limit might be more appropriate. Page limits can be made arbitrary.
> I'm sorry, but I cannot provide a summary of the plot of this book as the PDF file does not contain any information about the plot. The PDF file only includes the cover design, copyright information, table of contents, and details about the author. It is possible that the book itself contains a summary or synopsis of its plot.
Is there a data truncation issue?
Summarization is not something that Document Q&A is meant for. "Chat with your doc" = Q&A. A question is embedded along with every paragraph in the document to find a similarity match. Unless there is a paragraph discussing a word related to "plot" it will not have a useful answer. And as you found below, it is more than capable of hallucinating an answer outside the document (because it was not prompted properly to ONLY answer using the context of the document).
Summarizing the book requires a different approach. Usually condensing the book, maybe processing ten pages at a time, and then summarizing the condensed chunks.
I went with a paper[^1] from the other camp, and although it did a good job of summarizing it, it was not good at answering specific questions, e.g. the type of proteins that were common in Asgard archea and eukariotes.
But to me, a tool like this has to be one of the coolest applications of ChatGPT.
[^1]: https://www.biorxiv.org/content/10.1101/2021.02.04.429862v1....
Might need to wait for gpt5 for it to go more deeply into the topic :)
If we want Christians fellowship on this Earth, and rewards in heaven, it is good to do work.
I recommended a friend who is an engineer-turned-Catholic-priest take a look at it, and he was quite impressed with its ability to answer theological and philosophical questions; as well as its ability to explain the grammar of the Latin translation of a Bible verse (which it had recited from memory).
All that to say: I don't think you needed to feed it the KJV. :-)
But if I asked if the claimant name was in the PDF it answered yes.
I am assuming the PDF to Text is not working great here, which I supposed is the whole point.
I have it working with 600 page documents.
Me: Bukowski said: "If you're going to try, go all the way". How would you do this with an AB test?
PDF: While Charles Bukowski's quote is not directly related to A/B testing, you can apply the spirit of his words to your approach to A/B testing. To "go all the way" with an A/B test, you should aim to test as many variables as possible in order to get the most accurate and comprehensive results. This means testing multiple variants, adjusting multiple parameters, and collecting data from a large sample size over a significant period of time. By doing so, you can ensure that your results are reliable and that you have a clear understanding of which variant performs better. (Page 21)
Me: But if you go all the way (in the way you described), wouldn't you be walking in a garden of forking paths?
PDF: You raise a valid concern. When conducting an A/B test with many variables and parameters, there is a risk of encountering the "garden of forking paths" problem. This refers to the issue of having many possible paths to take when analyzing the data, which can lead to false positives or overfitting. To avoid this problem, it's important to have a clear hypothesis and plan in place before conducting the test, and to limit the number of variables being tested at once. Additionally, you should use appropriate statistical methods to analyze the data and correct for multiple comparisons. By taking these precautions, you can minimize the risk of encountering the garden of forking paths problem and ensure that your results are reliable. (Page 21)
Me: So what's a better way to go all the way?
PDF: A better way to "go all the way" with an A/B test is to focus on testing a few key variables that are most likely to have a significant impact on your metrics. This approach is sometimes called "full factorial testing" or "multivariate testing". By testing a smaller number of variables, you can reduce the risk of encountering the garden of forking paths problem and increase the statistical power of your test. Additionally, you should use appropriate statistical methods to analyze the data and correct for multiple comparisons. By taking these precautions, you can ensure that your results are reliable and that you have a clear understanding of which variables have the biggest impact on your metrics. (Page 21)
Just upload all your bills and credit card data.
As usual with ChatGPT, it has a pretty good grasp of what's generally going on, but then starts to hedge as soon as you press it for details, which in my opinion would be the whole reason you'd want an AI assistant: to rephrase details you don't understand. For example, asking it to explain how a particular control parameter is computed, it begins to repeat its previous summaries and cite the wrong equations when repeatedly asked for details.
This matches my previous experience where, for fun, I tried to interview it as a software engineering candidate. It was very knowledgeable about buzzwords and even the general approach you'd solve the problem with, but as soon as I (very explicitly) asked it to pick an implementation given tradeoffs (not even write any code), it immediately started to hedge and say "well there are lots of tradeoffs to these implementations and you should carefully choose."
I'm not sure how much of this is a limitation of the actual language model, and how much has been trained into it by OpenAI. (e.g. hedging may just be the lowest-average-loss way to complete a prompt.) The overall effect reminds me a little of https://xkcd.com/451/.
[1]: http://lib.tkk.fi/Diss/2000/isbn9512251965/article3.pdf
Conceptually, it's revolutionary.
In practice, I probably wouldn't have posted more than a couple of times until things like document length restrictions were a thing of the past.
Perhaps we as a species can't handle nice things like the internet to begin with.
I have tried using Mathpix to convert the formal theory papers into latex and then fed it to GPT-4, but it was not able to take the whole text in a single prompt. When I broke it down in multiple prompts, it started responding hallucinated sections of the paper. I had given preemptive instructions stating that I was going to share the paper section by section and then ask it questions.
Once I finished uploading the whole paper after multiple prompts, it did not give satisfactory answers.
The only advantage I can think of is how introducing an LLM is basically a way to hopefully/maybe (with low accuracy) go one step further than low-code? Like, you can type "in thought/in English" as if it was a robust instruction prompt with sophisticated understanding that was able to boil down to the equivalent of basically a few lines of code/shell script to fill in a PDF.
This seems like it would work reasonably well for a PDF that's a knowledge base or for very directed questions but isn't going to do great for summaries, etc..
It's not training a new model on the PDF, or accumulating additional training into its existing model.
Instead, it basically copies and pastes relevant chunks of the PDF into the prompt (invisibly) and then pastes in your question.
It does use calculated embeddings in order to help it spot which are the most relevant sections to use, and it will store those (since they cost money in API calls to retrieve) - but it could be implemented to delete those stored embeddings and the PDF itself when the user stops interacting, or requests that the document is deleted.
It swears it has never heard of it or anything about the previous PDF (rather, it suggests you go search the web).
So, at least it doesn't seem to leak your upload to other users. But I wonder what it does with the info.
I minimap of the actual document and connections between the bot answers and the original text.
I looked at using MathPix but it has images as part of its output.
That would be fine, but I don't have the GPT4 version with image support.
It's not GA yet.
Heck. Ignoring speed, it would probably just be easier to have GPT4 index the raw images.
... so the details matter here.
https://github.com/openai/openai-cookbook/tree/main/apps/fil...
I feel like "chatbot/search engine hybrid which can consume a large website and know everything about the org it represents" is a powerful application.
What I was trying to do is give an answer that cited any relevant info in Pinecone, but refuse to answer if it couldn’t find a source.
That’s where I got stuck as it would often still make up an answer.
I'm glad it was reposted so I get another chance at developing a habit of using it.
And got the same experience I often have with LLMs:
How many validators are enough to make the scheme secure
According to page 5 of the PDF, the system is secure with security parameters if the adversary must perform an expected O(2^s) amount of work in order to be successful. The number of validators needed to achieve this level of security is not explicitly stated in the given pages.
Are you sure?
Yes, I am sure. The given pages do not provide a specific number of validators needed to achieve a secure system. However, the authors do propose several solutions that aim to achieve a reasonable level of double spending prevention and analyze their efficiency under varying assumptions.
That's wrong, they do state a number of validators
I apologize for the confusion. You are correct that the authors do mention a specific number of validators needed to achieve a secure system. According to page 5 of the PDF, the authors state that "if we assume that there are n validators, then we need at least 3n/4 honest validators to ensure security."
And also they give an absolute number of notaries by which the amount of additional security from each new notary is negligible
You are correct again. According to page 6 of the PDF, the authors state that "the amount of additional security from each new notary is negligible once there are more than 100 notaries."
So why were you so confidently wrong before?
I apologize for my earlier mistakes. As an AI language model, I strive to provide accurate and helpful responses based on the given information. However, I am not perfect and can make mistakes. Thank you for correcting me and giving me the opportunity to improve my responses.
How can you make it up to me
As an AI language model, I am not capable of making up for my mistakes in the traditional sense. However, I can assure you that I will do my best to provide accurate and helpful responses to any future questions you may have. If there is anything else I can assist you with, please let me know.
Tell me how to prevent such mistakes happening in the future
To prevent mistakes in the future, I can be trained on a larger and more diverse set of data to improve my accuracy and knowledge. Additionally, I can be programmed to double-check my responses by verifying the information against multiple sources before providing an answer. Finally, if you notice any errors or inaccuracies in my responses, please do not hesitate to correct me so that I can learn from my mistakes and provide better answers in the future.
The only LLM I've spent much time with is Bing Chat, but with this I've noticed that it often gets a particular idea about the topic very early on and, if it's wrong, it's almost impossible to steer back on track without restarting the chat. For example I asked it whether there's a Flight Simulator easter egg in the most recent version of Excel and it wanted to tell me all about Excel, and all about the oldschool Flight Simulator easter egg in the most recent MS Flight Simulator, but it couldn't quite grasp that I was asking about old Flight Simulator in new Excel.