Comment by v3ss0n | Hacker News Reader

Comment by v3ss0n | Hacker News Reader

Parent Full thread

At this point , due to non-deterministic nature and hallucination context engineering is pretty much magic. But here are our findings.

1 - LLM Tends to pick up and understand contexts that comes at top 7-12 lines.Mostly first 1k token is best understood by llms ( tested on Claude and several opensource models ) so - most important contexts like parsing rules need to be placed there.

2 - Need to keep context short . Whatever context limit they claim is not true . They may have long context window of 1 mil tokens but only up to avg 10k token have good accuracy and recall capabilities , the rest is just bunk , just ignore them. Write the prompt and try compressing/summerizing it without losing key information manually or use of LLM.

3 - If you build agent-to-agent orchestration , don't build agents with long context and multiple tools, break them down to several agents with different set of tools and then put a planning agent which solely does handover.

4 - If all else fails , write agent handover logic in code - as it always should.

From building 5+ agent to agent orchestration project on different industries using autogen + Claude - that is the result.

Based on my testing the larger the model the better it is at handling larger context.

I tested with 8B model, 14B model and 32B model.

I wanted it to create structured json, and the context was quite large like 60k tokens.

the 8B model failed miserably despite supporting 128k context, the 14b did better the 32B one almost got everything correct. However when jumping to a really large model like grok-3-mini it got it all perfect.

The 8B, 14B, 32B models I tried were Qwen 3. All the models I tested I disabled thinking.

Now for my agent workflows I use small models for most workflow (it works quite nicely) and only use larger models when the problem is harder.

That is true too. But I found Qwen3 14B with 8bit quant fair better than 32B with 4b quant . Both kvcache at 8bit. ( i enabled thinking , i will try with /nothink)

I have uploaded entire books to the latest Gemini and had the model reliably accurately answer specific questions requiring knowledge of multiple chapters.

I think it works for info but not so well for instructions/guidance. That's why the standard advice is instructions at the start and repeated at the end.

I wonder if the serial-position effect is happening with LLMs.

https://en.wikipedia.org/wiki/Serial-position_effect

Something like it definitely, though not exactly. We also know that recall improves with proximate position of bits within the context.

Adherence to context is lossy in a way reminiscent of human behavior but also different in crucial ways.

Or under the covers are just putting all the text you fed at into a rag database and doing embedding search define route and snippets and answer your questions when asked directly. Which is the difference approach than recalling instructions

That’s pretty typical, though not especially reliable. (Allthough in my experience, Gemini currently performs slightly better than ChatGPT for my case.)

In one repetitive workflow, for example, I process long email threads, large Markdown tables (which is a format from hell), stakeholder maps, and broader project context, such as roles, mailing lists, and related metadata. I feed all of that into the LLM, which determines the necessary response type (out of a given set), selects appropriate email templates, drafts replies, generates documentation, and outputs a JSON table.

It gets it right on the first try about 75% of the time, easily saving me an hour a day - often more.

Unfortunately, 10% of the time, the responses appear excellent but are fundamentally flawed in some way. Just so it doesn't get boring.

Try reformatting the data from the markdown table into a JSON or YAML list of objects. You may find that repeating the keys for every value gives you more reliable results.

Thanks for the suggestion! I’ll start benchmarking my current md table setup against one using YAML. It's apparently slightly less verbose than JSON.

Gemini does a lot better at long context.

I wonder if those books were already in the training set, i.e. in a way "hardcoded" before you even steered the model that way.

Should be easy to test: ask the question without the book in the context window, ask again with the book in the context window.

Try multi-turn and agent-to-agent , it will breakdown , but Gemini is a lot better at larger context.

Mind if I ask how you’re doing this? I have uploaded short stories of <40,000 words in .txt format and when I ask questions like “How many chapters are there?” or “What is the last sentence in the story?” it gets it wrong. If I paste a chapter or two at a time then ask, it works better, but that’s tedious…

Claude’s system prompt is SO long though that the first 1k lines might not be as relevant for Gemini, GPT, or Grok.

Reply on news.ycombinator.com