Hacker News Reader

Although Gemini accepts very long input context, I found that sending more than 512 or so words at a time to the LLM for "cleaning up the text" yields hallucinations. That's why I chunk the raw transcript into 512-word chunks.

Are you saying it works with 70B models on Groq? Mixtral, Llama? Other?

View on HN

When you did this, I am assuming you cut the audio off around 5 mins?

https://github.com/google-gemini/generative-ai-js/issues/269...

Yeah, I've had no issues sending tokens up to the context limit. I cut it off with a 10% buffer but that's just to ensure I don't run into tokenization miscounting between tiktoken and whatever tokenizer my actual LLM uses.

I have had little success with Gemini and long videos. My pipeline is video -> ffmpeg strip audio -> whisperX ASR -> groq (L3-70b-specdec) -> gpt-4o/sonnet-3.5 for summarization. Works great.

Reply on news.ycombinator.com