Hacker News Reader

Comment by sqs | Hacker News Reader

I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing to search everything exhaustively). Happy to chat with anyone who's building this kind of thing, including with folks at Val Town, which is awesome.

View on HN

And when you're ready to do indexed search, Zoekt (over which Sourcegraph graciously took maintainership a while ago) is the best way to do it that I've found. After discounting both Livegrep and Hound (they both struggled to perform in various dimensions with the amount of stuff we wanted indexed, Hound moreso than Livegrep), we migrated to Zoekt from a (necessarily) very old and creaky deployment of OpenGrok and it's night and day, both in terms of indexing performance and search performance/ergonomics.

Sourcegraph of course adds many more sophisticated features on top of just the code search that Zoekt provides.

I've been surprised at how far you can get without indexing.

Ex. I always assume we'll need to add an index to speed up GritQL (https://github.com/getgrit/gritql), but we've gotten pretty far with doing search entirely on the fly.

What does 'on the fly' entail here?

I'm going to guess brute force - scan everything for the search term, rather than trying to use an index.

I'm always amazed at how fast ripgrep (rg) can brute force it's way through hundreds of MBs of source code.

Yes, exactly. When doing a search, we parse and search every file without any indexing.

Of course, it could still be sped up considerably with an index but brute force is surprisingly effective (we use some of the same techniques/crates as ripgrep).

I apply this thinking to lots of problems. Do the dumb thing that involves the least state and prove we need to lean more towards memory for speed. It’s much simpler to keep things correct when nothing is cached

You'll also be in a much better spot to pick appropriate indexing when you actually have sizable and representative workloads.

There was someone doing temporal databases that was compressing blocks on disk and doing streaming decompress and search on them. Things in L2 cache go very very fast.

I hope this and SCIP becomes a standard and we have more programming languages emitting symbols in SCIP format.

I thought SCIP got promoted into https://lsif.dev/ but chasing the https://github.com/sourcegraph/lsif-java link resolves to https://github.com/sourcegraph/scip-java so maybe I had the evolution relationship backward. Anyway, I'm thankful at least that code is still Apache 2

https://github.com/topics/lsif may interest this audience, too, since the scip topic tag seems to clash with something else

Also, I learned last night that GitLab embraces LSIF, too https://docs.gitlab.com/ee/topics/autodevops/stages.html#aut...

SCIP is an evolution of LSIF, basically: https://sourcegraph.com/blog/announcing-scip

My, and my friends experiences with SCIP indexers built by Sourcegraph have been less than stellar. They are buggy and sparsely maintained

I work at Sourcegraph and would love to learn more.

1. Which SCIP indexers did you having issues with?

2. What issues did you hit (can you share details or link to GitHub issues filed?)

Thanks!

Hello! I am Head of Engineering at Sourcegraph. I'd love to get feedback on which SCIP indexers you've had issues with, and, if you have the time, feedback on what sort of problems you've had with them. Thank you so much!

Hey guys, it's been over two months since I've been in the weeds with SCIP so I'm not going to be able to write very detailed issues, most of my experiences were with scip python and some in typescript.

1. roles incorrectly assigned to symbol occurences

2. symbols missing - this is a big one. I've seen many instances of symbols being included in "relationships" array that were not included in "symbols" array for the document, and vice versa. Plus "definition" occurrences have been inconsistent/confusing - only some symbols have those, and they don't always match where the thing is actually defined (file/position), and sometimes a definition occurrence has no counterpart in symbols array

3. the treatment of external packages have been inconsistent, they sometimes get picked up as internal definitions and sometimes not

I think SCIP is a great idea and I'd explore using it again if it got better. But I see that there are issues staying in the backlog for 6+ months which makes it seem from the outside like Sourcegraph is not prioritizing further development of scip

Thanks for the details. I appreciate you taking the time to give the feedback. We will use this to help to improve SCIP and the indexers.

Do you plan on ever allowing users to change the font size?

Any opinions on mozilla's DXR?

Reply on news.ycombinator.com