stephantul

It was directed at the parent who implied that we didn’t think about this.

I agree with your point about the evals and how you can get discontinuities: good search can be worse than bad search when agents can do many searches. We’re working on it

It's not probabilistic, and exact matches will always be preferred over non-exact. So if you search for a function name this will surface it.

This is a bit rude.

We didn't generate this project, we wrote it, a lot of it manually, and trained custom models. We'd been working in the real-time retrieval space for a while, and we thought coding was a good fit for this specific technology.

Oh sorry that happened. Feel free to open an issue or report it here

Wow awesome, thanks for sharing! This is really useful and very much like the experiments we want to be doing in the near future

This is true, agents just don't know a lot about the things they're looking at, e.g., the number of files, file sizes, etc.

Although for small codebases it also holds that whatever you would like to find it easy to find, so search still might help you with cost

Yeah I agree. I have used semble to quickly index a large monorepo and just ask a question about it, it surfaced the right files pretty quickly. Although without an IDE, it's difficult to display them in nice way

The comparison is in the benchmarks, see the README

For chunking Semble supports all languages supported by tree-sitter-language-pack. The models we train are trained on 6 languages, but can handle way more.

We hadn't found that one yet. Will do!

Yes, this is the main reason. We've released some rust stuff in the past, but Python is our main language

The comparison is with ripgrep, see the benchmarks.

Yeah we're also interested in doing this, it's on the roadmap together with optimization of the prompt and descriptions so that models have an easier time using it.

Perhaps anecdotally: we do use this tool ourselves of course, and it's been working pretty well so far. Anthropic models call it and seem to trust the results.

Even so. Take a look at the NDCG numbers for grep. It's not pretty

Hey, this is something we're actively investigating. We recently added a flag, `--include-text-files`, which, when set, also makes Semble index regular documents (i.e., markdown, text, json). This should also work relatively well.

The same holds for semble: the agent can fire off many different semble queries with different k/parameters.

I guess the point we’re trying to make is that you need fewer semble queries to achieve the same outcome, compared to grep+readfile calls.

You need readfile to do something with those tokens. Grep only gives you the matching lines, not the context.

1) yes! It’s not accuracy, but ndcg 2) we assume that if the agent gets the correct answer in the returned snippets it does not need to read further

Hey! Co-author here. The benchmark currently only measures retrieval accuracy.

We’re interested in measuring it end to end and also optimizing, e.g. the prompt and tools, for this, but we just haven’t gotten around to it.

How would you conceptualize recall in this case? Is searching through the current version of your code and possibly git history not enough?

Really cool! I was investigating PCA on retrieval, thanks for the references.

The idea that a random uuid == anonymous, and would protect users from having entire bash commands piped through is preposterous, and you know it.

It’s a serious comment, posted solely with the intent to sow doubt.

Ah ok sorry, it just looked like you were speaking on his behalf.

I’m not saying it invalidates the result. I am saying that they knew the headline and comparison was not correct and they still decided to roll with it. It’s an incorrect representation of what happened, designed to get eyeballs and possibly vc dollars.

Yes it matters, because it’s not a measurement of whether it accomplishes the task if a human tells it how to solve it.

Buddy… your son gets a top post on HN in which he clearly mentions you, yet you feel the need to make an account just to correct him in the first comment? Can’t you send him a message and let him correct it?

The fact that this was on the set of training problems with a custom harness basically makes the headline a lie.

What if you give opus the same harness? Do people even care about meaningful comparisons any more or is it all just “numbers go up”

Mannnn, here I thought this was going to be an informative article! But it’s just a commercial for the author’s consulting business.

Do you have benchmarks? Naively I would compare this to Numba? But maybe I am way off the mark here