Big Post About Big Context | Hacker News Reader

Big Post About Big Context

gonzoml.substack.com

49 pointsche_shr_cat19 comments

One point the article argues is that with 1M+ token context windows, we shouldn't use RAG to include chunks from a larger collection -- just include everything! This is a very important open question to me, as I have been building a fairly nuanced RAG pipeline for AI coding.

With GPT-4 Turbo, it is definitely NOT a good idea to just throw lots of extraneous code into the 128k content window. That distracts and confuses GPT, makes it code worse and less able to follow complex system prompts. You get much better results if you curate a smaller portion of the code that is relevant to the task at hand.

I am really interested to find out if Gemini has these same tendencies. If so, quality RAG is going to remain valuable for curating the context. If not, then Gemini would have a huge advantage over GPT-4. It would be really valuable to be able to naively harness such a large context window.

There is a lot of emerging evidence that ability to reason, and ability to compare within-context dramatically declines with large context.

It is unable to find edits: https://twitter.com/mengdi_en/status/1763338245696274469

It hallucinates changes badly: https://twitter.com/thomasahle/status/1763408041960231010

To be fair, finding diffs is an odd way to use LLMs, but does show the highly variable abilities of these models over bigger contexts.

In other models, reasoning performance declines with more tokens (no data yet here for Gemini 1.5 Pro): https://arxiv.org/html/2402.14848v1

Note in that last chart that significant declines in reasoning are apparent over just a few thousand tokens.

I took the text of Macbeth and changed the dialect of one line:

Original: "The rest, that are within the note of expectation, alreadie are i'th' Court"

New: "It looks like everyone is here already"

Then I gave the entire text to Gemini, ~400000 tokens, and asked it to analyze the play's diction and style, and to identify any lines that didn't match the prevailing dialect.

It found that one line I'd changed.

Isn’t it quite likely it already had the whole text from its training data? Which might make it a bit of a different case.

I was responding to the assertion that the "ability to compare within-context dramatically declines with large context".

My test demonstrates that even with a large context, Gemini 1.5 finds the needle in the haystack.

And it's not even searching for specific terms or meanings, the only clues are subtle differences in style and diction.

I'm not surprised, those documents probably contain a lot of information which causes the model to get confused/lost/etc.

It also seems to line up with random paragraph location doing the worst in that paper.

If you think about it, RAG is a relatively primitive “first pass attention layer” that is binary and semi-heuristic based. I think it’s fairly safe to say that in the long term RAG will be integrated into the model architecture somehow, just a matter of when :)

Cosign, I feel like the longer context window is mostly a dud, especially above 16K tokens.

I don't miss sweating the details on what exactly to fit into 4096, but if I just use GPT-4, I'm left with the distinct impression the current context extension stuff is hacky, and something needs to be done in training to actually make use of it.

Google miraculously getting 1M context with perfect recall is more fuel for my "don't tell me, show me" fire when it comes to Google. They're batting roughly 0 for 2 years now

The main thing I'd like large context for is the ability to shove a ton of API/Python/etc documentation into the window for a given problem so it knows better what to do.

In terms of the actual output it can be annoying if the convo goes on too long and it hits the window, but for technical things this hardly matters as you can just restart it - more of an RP issue as i understand.

Also, while the cost of tokens is lower, let's argue it's cheap enough not to care. Reading 1m tokens surely isn't realistic for latency?

If sub-quadratic architectures (eg Mamba) become a thing, it will become feasible to precompute most of the work for a fixed prefix (i.e. system prompt) and the latency can be pretty minimal. Even with current transformers, if you have a fixed system prompt, you can save the KV cache and it helps a lot (though the inference time of each incremental token is still linear).

I think this kind of haystack recall is going to be extremely valuable to large enterprises with massive volumes of standards/practices/technical documents.

The status quo is that you have employees ingest and internalize this material over years of employment, then have those experienced employees manage large projects by telling the project if they're in compliance with the organization's standards.

A reliable recall system like this isn't just a drop in replacement for new employees, its a drop in replacement for a lot of what makes the expensive employees important: the backwards-and-forwards knowledge of the standards/practices.

Although our standard template constructors store all of mankind's knowledge and obviate the need for dangerous thinking, only a tech-priest of great experience may adequately consecrate them.

Is this a reference or just some solid on the fly sci-fi commentary?

It's a reference to WH40k, a fictional universe where faster-than-light travel survived the collapse of civilization in the form of pragmatic tools called "standard template constructors," made to take the burden of science and engineering staff off the shoulders of small planetary colonies, and the formalized rituals of the post-apocalyptic martian cults that remembered in some ways how to use them.

can someone explain me, maybe with an example, what this part means:

> 6) A very important and at the same time difficult class of solutions is model result validation

Thanks.

That means the models still confabulate and make other errors, and for many cases it's a problem, so there should be solutions to control model output quality

I updated the post a bit, added a few clarifications

the close example is Machine translation quality estimation (MTQE)

Reply on news.ycombinator.com