Hacker News Reader

Comment by aliljet | Hacker News Reader

There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.

View on HN

Considering the state of the field ( RAG/retrieval/evaluation) I have 0 trust in it, even more if it's closed source with bullshit claim like that.

Everything is vibe sloped to death, and dead after a few months to a couple of years (and not hard to be 100 cheaper than GPT-5.6 sol ... DS is basically free and I guess already 100 times cheaper or more, and here another slope ).

castform founder here. we should have made it more prominent on the blog but here's the full code example: https://github.com/castform-ai/benchmax/tree/main/examples/n...

Are you using blind chunking or section aware chunking?

for the example here the chunking is section aware -> but the general training data synthesis pipeline is agnostic to type of chunking

if chunking is section aware, how do you manage large section embeddings?

Thank you for the github link, what is the license? I went briefly over several repo, unless I missed it somewhere the license was often missing.

it's apache 2.0: https://github.com/castform-ai/benchmax/tree/main?tab=Apache...

I'm curious about this too as I've been working with a 1300 page document on procedures for [industry]. Where every single section is primarily about [industry] with minor differences in verbs actions etc for the procedures.

I've found with traditional embeddings that obviously you're getting an average of the content of the chunk even with the semantic awareness magic. And our (or I guess the) core problem seems to be a lack of enough contrast between chunks with makes one-shot pure embedding based RAG extremely difficult and low quality.

I use detailed project files. It has data regarding the project and subtasks as well as task status. It doesn’t depend on agent context and it’s managed to keep the agent on track. Feature creep with the new models is a very real issue. Capturing principles and how to reconcile tasks helps too. Even today it brought up a source of truth issue it had detected. There were multiple authorities born out of a patch and it used that principle to highlight and resolve the problem.

<founder of castform here> tldr: we generated synthetic training questions from the gitlab product handbook.

totally agree that this larger corpus with harder to search information would be a good way to stress test - i'm sure we will encounter more interesting problems to solve. love to hear any suggestions of corpus to search against that is not just the public internet

the training run link is also a little buried but here, you can see the comparison against the various models and their exact traces: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

Reply on news.ycombinator.com