Comment by FooBarWidget | Hacker News Reader

Comment by FooBarWidget | Hacker News Reader

In my opinion a major obstacle right now is the token limit. This makes it very cumbersome to operate on large input documents, e.g. summarizing a large document. It doesn't help that the limit includes the response as well.

Prompt engineering helps me better understand the limits and ways to work around them. But it's still painful.

Today I tried summarizing a long video (~1 hour) using Videohighlight but their AI quickly reached the token limit.

I also tried to use ChatGPT to (somewhat ironically) translate a new AI chat regulation proposal by the Chinese government, for which the government asks public feedback. http://www.cac.gov.cn/2023-04/11/c_1682854275475410.htm I used prompt engineering to get a translation quality and readability better than what I can get through Google Translate. This isn't a short document but it's not very long either. I had to break it into three in order to get ChatGPT to translate it all.

In case anyone is curious about the final translation: https://gist.github.com/FooBarWidget/201ea5e0983d05d21f6719b... This is the prompt I used, incorporating prompt engineering lessons I've learned. I assigned a role to the LLM, provided appropriate context, and provided constraints for its output.

> You are a Chinese-to-English translator with decades of experience translating Chinese government policy documents to an audience that's not familiar with Chinese governance, nor with the way that the Chinese government writes. What follows is a part of a draft policy document on potentially regulating AI chat services. The Chinese government has asked the public for feedback.

> Begin introduction:

> ...

> End introduction.

> Begin part of policy document body:

> ...

> End part of policy document body.

> Translate the the policy document body part to English. Do not translate the introduction. Write in a manner that's easily readable for an audience that has no experience with Chinese policy, Chinese idioms or the way the Chinese government writes.

Very interesting use of ChatGPT and prompt engineering. For your problem, summarizing a large document, splitting the document into smaller parts is indeed the way to go. I also had problems myself with operating on large documents. In my case, I had an insurance policy that I wanted to extract information from.

My solution: use the OpenAI API to convert the document to OpenAI's embeddings and saving those embeddings to a vector database. Then, use similarity search on the database to find chunks of the document that might be related to my query and pass only those chunks to GPT for the information extraction prompt.

I plan to create a guide on how to tackle these problems after I consolidate my findings.

I'm increasingly seeing these kinds of solutions for similar tasks. I wonder if we are seeing the discovery of new abstractions from using LLMs.

Very cool solution!

My solution was to write a bit of code that writes a CSV, then I used a langchain-based CSV agent. Since that one calls on pandas it effectively has no token limit, but it also has no overview of the data, only what pandas tells it.

this is how we enable 10mb+ file ingestion and search /w http://mixpeek.com

Also doing this, works really well. Check out James Briggs on YouTube. Excellent tutorials on how to achieve this.

Can you refer to a specific video?

https://youtu.be/tBJ-CTKG2dM

Goes into workings of retrieval augmentation with example

I've actually coded something like this. The trick is to split it in a smart way. What I did is the following:

1. Download the video 2. Split it in max length for whisper 3. Used whisper for transcription on the chunks 4. Used gpt for cleaning the transcription 5. Unioned the cleaned transcribed chunks 5. Used the langchain "refine" function of the summary chain

It is quite slow and expensive because the refine summary is doing a lot of calls, but the result is amazing. And you don't need a vector DB for that because the summary is serial.

For Q&A tho, an embeddings DB is a must

If telling Chat GPT that it has “decades of experience” leads to better results, why not say centuries of experience instead?

If you don’t know how every token of input affects the output, is engineering an accurate way to describe what you’re doing?

I don't think the training data has works of centuries-old translators.

> If you don’t know how every token of input affects the output, is engineering an accurate way to describe what you’re doing?

We in fact don't know exactly. Prompt engineering seems to be a bag of tricks that change the probabilities of outputs. But I didn't invent the term.

So that prompt should fail then, right?

Prompt success is not binary as there are multiple ways of being wrong as well as multiple ways of being right.

I don't think this is a strong argument against calling it "prompt engineering".

Engineers have built systems based around things they didn't fully understand for centuries.

We have learned an enormous amount about materials engineering and metallurgy since building the Brooklyn Bridge for example.

If you don't fully understand how a system you are building on top of works, the engineering approach is to methodically experiment. That's what prompt engineering is.

(One argument that works for not calling prompt engineering "engineering" is to point out that in many disciplines engineering requires certifications and licensing - the same reason people sometimes argue against "software engineering" as an engineering discipline.)

I've never heard the licensing argument come up about 'reverse engineering'.

edit: Which I feel is closer to prompt engineering than software engineering.

Because ChatGPT is pulling from the collective human digital linguistic consciousness and humans don't truthfully mention having centuries of experience. Whereas humans do talk about having decades of experience. Similar to how you could ask about near-death experiences but not about what it's like to have been dead for 5 years: There's no way for truthful information about that to reside in the dataset that the LLM is pulling from.

For the same reason you're more likely to take me seriously if I say I have "decades of experience" versus "centuries of experience".

Texts where someone has "centuries of experience" are likely to be fantasy or sci-fi, which bias against reliability.

Progressive summarization works pretty well - I'm using that for https://findsight.ai

You can even get lesser LLMs to do the bulk reduction that have GPT clean it up on the way to even less content. Admittedly, that does take a lot of prompt engineering, chunk selection and reinforcement though (LLM supervising LLM).

I understood everything except:

> reinforcement though (LLM supervising LLM).

Is there something I can read to understand what that looks like?

I don't think this approach is formalized but I can give a few examples:

A) Prompt leak prevention: chunk and embed LLM responses, than compare against original prompt to filter out chunks that leak the prompt

B) Automatic prompt refinement: Prompt a cheap model, use an expensive model to judge the output and rewrite the prompt (this is in part how Vicuna[1] did eval for their LLaMa fine-tuning)

Basically using LLMs in the feedback loop.

[1] https://vicuna.lmsys.org

Try Universal Summarizer. No token limit.

https://kagi.com/summarizer

And this is even worse on locally ran models. I get 2k tokens of context, max.

Reply on news.ycombinator.com