Hacker News Reader

RAG is taking a bunch of docs, chunking them it to text blocks of a certain length (how best todo this up for debate), creating a search API that takes query (like a google search) and compares it to the document chunks (very much how your describing). Take the returned chunks, ignore the score from vector search, feed those chunks into a re-ranker with the original query (this step is important vector search mostly sucks), filter those re-ranked for the top 1/2 results and then format a prompt like;

The user ask 'long query', we fetched some docs (see below), answer the query based on the docs (reference the docs if u feel like it)

Doc1.pdf - Chunk N Eat cheese

Doc2.pdf- Chunk Y Dont eat cheese

You then expose the search API as a "tool" for the LLM to call, slightly reformatting the prompt above into a multi turn convo, and suddenly you're in ze money.

But once your users are happy with those results they'll want something dumb like the latest football scores, then you need a web tool - and then it never ends.

To be fair though, its pretty powerful once you've got in place.

Is RAG how I would process my 20+ year old bug list for a piece of software I work on?

I've been thinking about this because it would be nice to have a fuzzier search.

Yes and no, for human search - its kinda neat, you might find some duplicates, or some nearby neighbour bugs that help you solve a whole class of issues.

But the cool kids? They'd do something worse;

They'd define some complicated agentic setup that cloned your code base into containers firewalled off from the world, give prompts like;

Your expert software dev in MY_FAVE_LANG, here's a bug description 'LONG BUG DESCRIPTION' explore the code and write a solution. Here's some tools (read_file, write_file, ETC)

You'd then spawn as many of these as you can, per task, and have them all generate pull requests for the tasks. Review them with an LLM, then manually and accept PR's you wanted. Now your in the ultra money.

You'd use RAG to guide an untuned LLM on your code base for styles and how to write code. You'd write docs like "how to write an API, how to write a DB migration, ETC" and give that as tool to the agents writing the code.

With time and effort, you can write agents to be specific to your code base through fine tuning, but who's got that kind of money?

You'd be surprised how many people are actually doing this exact kind of solutioning.

It's also not that costly to do if you think about the problem correctly

If you continue down the brute forcing route you can do mischievous things like sign up for thousands and thousands of free accounts across numerous network connections to LLM APIs and plug away

I feel called out, lmao. I’m building an agentic framework for automated pentesting as part of an internal AppSec R&D initiative. My company’s letting me run wild with infrastructure and Bedrock usage (bless their optimism). I’ve been throwing together some admittedly questionable prototypes to see what sticks.

The setup is pretty basic: S3 for docs and code base, pgvector on RDS for embeddings, Claude/Titan for retrieval and reasoning. It works in the sense that data flows through and responses come out… but the agents themselves are kind of a mess.

They think they’ve found a bug, usually something like a permissive IAM policy or a questionable API call, and just latch onto it. They tunnel hard, write up something that sounds plausible, and stop there. No lateral exploration, no attempt to validate anything in a dev environment despite having MCP tools to access internal resources, and definitely no real exploitation logic.

I’ve tried giving them tools like CodeQL, semgrep and Joern, but that’s been pretty disappointing. They can run basic queries, but all they surface are noisy false positives, and they can’t reason their way out of why it might be a false positive early on. There’s no actual taint analysis or path tracing, just surface-level matching and overconfident summaries. I feel like I’m duct-taping GPT-4 to a security scanner and hoping for insight.

I’ve experimented with splitting agents into roles (finder, validator, PoC author, code auditor, super uber hacker man), giving them memory, injecting skepticism, etc., but it still feels like I’m missing something fundamental.

If cost isn’t an issue, how would you structure this differently? How do you actually get agents to do persistent, skeptical, multi-stage analysis, especially in security contexts where you need depth and proof, not just plausible-sounding guesses and long ass reports on false positives?

Seems like you need a way to dictate structured workflows, in lieu of actually being able to train them up as soc analyst. Sounds like a fun problem!

You could try just exporting it as one text or XML file and seeing if it fits in Genini's context

I don't think it will. Gemini Pro has a context window of 2 million tokens which they say translates to around 1.5 million words. We have on the order of 100,000 logged issues and a typical issue description is around 500 words.

Or you find your users search for id strings like k1231o to find ref docs and end up needing key word search and reranking.

Sorry for my lack of knowledge, but I've been wondering what if you ask a question to the RAG, where the answer to the question is not close in embedding space to the embedded question? Will that not limit the quality of the result? Or how does a RAG handle that? I guess maybe the multi-turn convo you mentioned helps in this regard?

The way I see RAG is it's basically some sort of semantic search, where the query needs to be similar to whatever you are searching for in the embedding space order to get good results.

I think the trick is called "query expansion". You use an LLM to rewrite the query into a more verbose form, which can also include text from the chat context, and then you use that as the basis for the RAG lookup. Basically you use an LLM to give the RAG a better chance of having the query be similar to the resources.

Thanks for the answer! I think you are right, I've also heard of HYDE (Hypothetical answer generation), that makes an LLM encode a guess as the answer into the query, which may also improve the results.

> So what is is marketing stuff about "directly into a model's working memory."? Is my mental view wrong?

Context is sometimes called working memory. But no your understanding is right: find the right document through cosine similarity (and thus through embeddings), then add the content of those docs to the context

One of the things I find confusing about this article is that the author positions RAG as being unrelated to both context engineering and vector search.

The directly into working memory bit is nonsense of course, but it does point to a problem that is probably worth solving.

What would it take to make the KV cache more portable and cut/paste vs. highly specific to the query?

In theory today, I should be able to process <long quote from document> <specific query> and just stop after the long document and save the KV cache right? The next time around, I can just load it in, and continue from <new query>?

To keep going, you should be able to train the model to operate so that you can have discontinous KV cache segments that are unrelated, so you can drop in <cached KV from doc 1> <cached KV from doc 2> with <query related to both> and have it just work ... but I don't think you can do that today.

I seem remember seeing some papers that tried to "unRoPE" the KV and then "re-RoPE" it, so it can be reused ... but I have not seen the latest. Anybody know what the current state is?

Seems crazy to have to re-process the same context multiple times just to ask it a new query.

would loading the KV cache from disk be faster than just recomputing it?

imo the discontinuous segments bit would not work because of the causal dependence in transformers + RoPE as you mention, but maybe could be possible

Do you have any links to the papers for the “unRoPE” and “re-Rope” technique? I tried some searching and couldn’t find anything. I would love to look into this idea more.

I think that copy/paste-able KV cache idea sounds pretty promising. It might lose some of the inter-document context and attention that would get built up in the hidden state of the model as it processes the prompt. Maybe just throw in some ‘reasoning’ tokens before it gives its answer to give it a chance to attend cross-document

> In theory today, I should be able to process <long quote from document> <specific query> and just stop after the long document and save the KV cache right?

People do this, it's called prefix caching.

There's also https://arxiv.org/abs/2506.06266 where they compress the context down to a smaller representation they call a "cartridge," and composing cartridges from different contexts seems to work reasonably well.

Perhaps the person that wrote it is also confused. I guess Geminis embedding model offers multilingual support, but we can use anything. The assumption is the developer uses these embeddings on their end with their implementation of storage/querying (their own vector db). The confusing thing is the article is suggesting that whole process is now done automatically soon as you send the embeddings to Gemini (which doesn’t even make sense, shouldn’t it only take text?).

Your mental model is correct.

They're listing applications of that by third parties to demonstrate the use-case, but this is just a model for generating those vectors.

At least in theory. If the model is the same, the embeddings can be reused by the model rather than recomputing them.

I believe this is what they mean.

In practice, how fast will the model change (including tokenizer)? how fast will the vector db be fully backfilled to match the model version?

That would be the “cache hit rate” of sorts and how much it helps likely depends on some of those variables for your specific corpus and query volumes.

> the embeddings can be reused by the model

I can't find any evidence that this is possible with Gemini or any other LLM provider.

Yeah given what your saying is true and continues to be,

Seems the embeddings would just be useful for a “nice corpus search” mechanism for some regular RAG.

LLMs can’t take embeddings (unless I’m really confused). Even if it could take embeddings, the embeddings would have lost all word sequence and structure (wouldn’t make sense to the LLM).

This can’t be what they mean. Even if this was somehow possible, Embeddings lose information and are not reversible, I.e embeddings do not magically compress actual text into a vector in a way that a model can implicitly recover the source text from the vector.

LLMs can use search engines as a tool. One possibility is Google embeds the search query through these embeddings and does retrieval using them and then the retrieved result is pasted into the model's chain of thought (which..unless they have an external memory module in their model, is basically the model's only working memory).

I'm reading the docs and it does not appear Google keeps these embeddings at all. I send some text to them, they return the embedding for that text at the size I specified.

So the flow is something like:

1. Have a text doc (or library of docs)

2. Chunk it into small pieces

3. Send each chunk to <provider> and get an embedding vector of some size back

4. Use the embedding to:

4a. Semantic search / RAG: put the embeddings in a vector DB and do some similarity search on the embedding. The ultimate output is the source chunk

4b. Run a cluster algorithm on the embedding to generate some kind of graph representation of my data

4c. Run a classifier algorithm on the embedding to allow me to classify new data

5. The output of all steps in 4 is crucially text

6. Send that text to an LLM

At no point is the embedding directly in the models memory.

You're right on this. "Advanced" RAG techniques are all complete marketing BS, in the end all you're doing it passing the text into the model's context window.

Your comment really helps me improve my mental model about LLM. Can someone smarter help me verify my understanding:

1) at the end of the day, we are still sending raw text over LLM as input to get output back as response.

2) RAG/Embedding is just a way to identify a "certain chunk" to be included in the LLM input so that you don't have to dump the entire ground truth document into LLM Let's take Everlaw for example: all of their legal docs are in embeddings format and RAG/tool call will retrieve relevant document to feed into LLM input.

So in that sense, what do these non-foundational models startups mean when they say they are training or fine tuning models? Where does the line end between inputting into LLM vs having them baked in model weights

(1) and (2) are correct (well, I don’t know specifics of Everlaw). Fine tuning is something different, where you incrementally train the model itself further using more inputs, so that given the same input context it will produce better output in your use case.

To be more precise, you seldom directly continue training the model, because it’s much cheaper and easier to add some more small layers to the big model and train those instead (see LoRA or Peft).

Something like Everlaw might do all three, by fine tuning a model to do better at discovery retrieval, then building a RAG system on top of that.

Oh what you don't understand is that LLMs also use embeddings inside, it's how they represent tokens. It's just that you don't get to see the embeddings, they are inner workings.