Amazing progress by open-source. However, the 64K input tokens and especially the 8K output token limit can be frustrating vs o1's 200K / 100K limit. Still, at 1/30th the API cost this is huge.
One of the cheaper Gemini models is actually only 8B and a perfect candidate for a release as a FOSS Gemma model but the Gemini 8B model contains hints of the tricks they used to achieve long context so as business strategy they haven't released it as Gemma FOSS model yet.
Prompt 1 (64k) CoT (32k) Answer 1 (8k)
CoT 32k context is not included in the 64k input. So it’s actually 64k + 32k + 8k.
Prompt 2 (32k) + Previous CoT 1 (32k - this time it will be counted because we are chaining and these are two different API calls) Answer 2 (8k)
Another way to optimize this is to use another model to pick up only the correct CoT from the current answer and pass that as CoT for the next prompt. (If you are feeling adventurous enough, you could just use R1 to select the correct CoT but I think it will go insane trying to figure out the previous and current CoT)