Datasette-ripgrep: a regular expression search engine for your source code
simonwillison.net
simonwillison.net
I built this partly as a learning exercise, to see how far I could push the idea of a Datasette plugin and to figure out how to call external processes from Python asyncio.
I also built it because I had a hunch that the "datasette publish" mechanism for deploying Datasette to serverless hosting providers such as Google Cloud Run could make it easy to deploy code search engines too. I think the most interesting thing about this project is the GitHub Actions workflow I wrote that deploys the demo instance running against code from 60+ repositories: https://github.com/simonw/datasette-ripgrep/blob/main/.githu...
I'm using it with my emacs (+helm), I just have a global keybinding which instantly opens a new code search. I could imagine having a web frontend being very convenient for people who aren't hooked on Emacs, even I often want to persist a code search result for several days, so will give datasette-ripgrep a try!
It's a regular expression search engine built on ripgrep that can search in virtually anything (pdf files, zip archives, movie subtitles, sqlite databases, png+ocr, ...).
There is always the case that, when trying to find all instances of some code, it will miss that one case which is not covered by the initial intuition.
For example, to find uses of the function "foo()", the first idea is usually to search for ".foo(". But that won't find places where the function is passed as value. Nor places where devs were inconsistent and wrote / didn't write a space between the name and the opening brace, like ".foo (" (this is a common style for Gnome or Glib code). Not to mention if the language is C++ and then you have to distinguish between ".foo", "->foo", and "::foo".
In conclusion, regex might be OK to have a superficial look at some code, but never as a reliable source of information, unless a lot of work and ironing out of corner cases has been invested in the regex itself, effectively considering them part of the code, with unit tests and all the fuss that it entails.
I wonder if the AST built by Tree-Sitter could also help with this type of search - does anyone know of any existing solutions for this?
https://semgrep.dev, though it's mostly an analysis tool it can be used as a search tool. IIRC it's not super fast, but for the cases where there is no way to really contort a regex into something suitable (the regex has way too many false positives and / or negatives) it works rather well.
I've been working on a trigram-based search engine with support for exactly this (via github.com/trishume/syntect) over the past several months and plan on open-sourcing it soon. Cool to see someone else with this idea!
You might also like https://comby.dev - it is aware of code structure without parsing files
Awesome, I look forward to that - you'll post a "show HN" for it, I hope? :) Does syntect's lack of support for the newest sublime-syntax features cause you any problems?
> You might also like https://comby.dev - it is aware of code structure without parsing files
Thanks for sharing, will check it out!
Show HN -> Yep :)
Lack of support for newest sublime-syntax features: less than you would expect, but a little for sure.
Most of the syntax definitions in the wild today don't really use the newer features, at Sourcegraph I wrote a little Rust HTTP server wrapping syntect[1] and we use it for all our syntax highlighting for the past several years, I would say it works on like 95% of code files even if you include a lot of additional syntaxes that are untested with Syntect[2]. That said, it does barf hard on some specific files - either taking a really long time to do the work or getting completely stuck in a busy waiting loop for some reason. That said, it's still the 2nd best syntax highlighter out there (second only to Sublime itself.)
One of my hopes for this side project is that I'll be able to contribute more time upstream with e.g. a more extensive test suite for Syntect against a much larger number of syntax definitions from the wild instead of just Sublime's built-in ones.
For example, you can type out "foo" at first which will, say, display 7 results, then when you add "()" you see the list shrunk by one, so you immediately know the regex missed one it's supposed to capture.
Also, there is a few lines trick [1] that will make it work with "M-x grep-find", respecting git project tree.
I really wish the git protocol had a way to perform remote git grep so you don't need to install potentially massive repos just to search for keywords.
Edit: Here's a live sandbox demo[1] of it hosted.
[0] https://ripgrep.datasette.io/-/ripgrep?pattern=%5C.plugin_co...
Is Kythe used to index some prominent open source project/s?
It’s also used to index Google’s open source: https://cs.opensource.google/search?q=Kythe&sq=
Thread from earlier this year: https://news.ycombinator.com/item?id=22551856
Not sure what’s giving you that “abandon ware” vibe. Lack of marketing materials? It’s pretty thorough documentation on the indexer.
Clicking "web ui" on the front page took me to this page [1] that is scant on details, linking to a caley release from a year ago, probably is was written long time ago and never updated (makes me think the rest of the pages may be the same). I can tell ease of deploying is not a priority too.
Just a little bit of css and linking to source.chromium.org/chromium and cs.opensource.google as example deploys would make the project look more alive.
My 2 cents :-)
OpenGrok is powered by Universal Ctags (https://github.com/universal-ctags/ctags) underneath and uses a Java servlet web server (Apache Tomcat) with search powered by Apache Lucene.
OpenGrok is older, boring technology but it works.
It is refreshing to see a new, Rust-powered approach to code search.
Edit: The discussion has taken a weird turn into support for ignore files. To be clear, this feedback is about dynamically filtering searches when using a website to do code search, most likely searching someone else’s repo on someone else’s website; you don’t have access to the filesystem, the search form is the only input. So all this “rg supports ignore files” (I’m aware, but thanks) is not really relevant.
I'm talking about excluding files and directories during searches, through a directive like, say, "-exclude tests -exclude test_*.py".
Sure, I guess technically you can add a bunch of additional patterns to gitignore before you use this web frontend, but that would be a really weird workflow, and it's impossible when the web frontend is run by someone else.
- Most of the time I don’t want to see search results in tests, but occasionally I actually want to find things in tests and not anywhere else. Different people have different filtering needs at different times, you can’t have a ignore file that satisfies every need. Search is about filtering to begin with.
Once you do cross repos search providing global filters on every query turns into an awful user experience.
WRT web frontends: datasette runs locally.
> Datasette is a tool you run against your stuff usually. ... datasette runs locally.
I don't think so? I could be wrong but I thought a major use case (or the major use case) is providing a frontend for your data to other people. Like on https://ripgrep.datasette.io/-/ripgrep.
To quote https://docs.datasette.io/en/stable/,
> Datasette is aimed at data journalists, museum curators, archivists, local governments and anyone else who has data that they wish to share with the world.
_I_ use datasette and I only ever used it locally or in a docker container. In either case for datasette-ripgrep to work you need to check out the repos you want to search in one folder that datasette will then invoke ripgrep on.
There are folks that publish datasette installations for public consumptions but even in that case I would assume that code search was preconfigured to make any sense with ignore files.
>> Datasette is aimed at data journalists, museum curators, archivists, local governments and anyone else who has data that they wish to share with the world.
That doesn't preclude installing the program locally.
The official "getting started" talks of remote / web datasettes as "demos" and "trials".
Of course not. But the project makes it pretty clear that it's designed with multiuser in mind. Can you host multiuser web apps only for yourself? Of course, I do that all the time. The unreasonable thing is saying "I use this thing exclusively for my own stuff on my local network, so screw your requests for multiuser, public-facing use cases."
It really doesn't. AFAIK Simon Willison (the author and creator) does his exploration locally, on things of interest to himself.
Publishing the analysis for others to see is no more "multiuser" than publishing a PDF.
What about the one-off search of someone else’s code? I don’t want to check out the repo just to add/edit a file so I can run ripgrep locally. If GitHub has a search feature that is entirely online, it would make sense for people to request they improve it.
So for instance in my case I have all the code I work with in ~/Development. Since I also use ripgrep for local development I typically include .ignore/.rgignore files to control what I'm searching around in my repos.
Doing "excludes"/"includes" across a variety of repositories is hard at query time because they layouts of those repositories can be very different. For instance in some JS repos you do want to search in "dist" whereas in others you don't because "dist" gets generated out of "src" for instance and just ends up being a duplicate etc.
The way I see doing includes/excludes at query time is this. You run a search without excludes for example. You see a bunch of stuff coming back all in one directory that you don't want for this particular search. So you add an exclude rule for that directory. And then just refine results that way.
If you have a bunch of similarly named directories, then I think that just means that the exclude rules at query time need to be more specific. e.g., use the full path to the directory.
I think the feature being requested here is an interactive feature of the user interface, where as ignore files are more like a static fact of a directory tree.
I think people are imagining that it should be the repo-author's responsibility to create the equivalent of an `.rgignore` file ahead-of-time (since they know the codebase's structure, they know what's useful vs. useless to show up in a search of the code — pretty objectively, "code" should show up, and "not-code" shouldn't); and that code-search on websites like Github should respect the rules in such a file if it were discovered in the repo.
If such a thing existed and was in wide use, you — a random passerby to the codebase, who doesn't yet understand it — wouldn't need to struggle to do any dynamic filtering, because static filtering would already have been done for you.
That's the thing, it's impossible to create a generic ignore list that caters to everyone's interest. Could the_mitsuhiko, the creator of flask, provide such a list for flask? Nope. Some people want to understand how register_blueprint is implemented and don't want to see anything from test, docs and examples. Some people want to learn how to use register_blueprint, so they specifically want to limit themselves to examples. Some other people want to check if there are test cases for a certain API. You can't ignore any file that may be interesting to someone, which is basically all of them.
ripgrep already does ignore certain files! Specifically, it ignores binary and hidden files, and files in your .gitignore. This is part of what makes ripgrep's default output so much more "readable" than GNU grep's default output: its few default filters mean that it's not matching from e.g. the contents of your .git/ directory, whereas GNU grep does show those matches by default.
> Could the_mitsuhiko, the creator of flask, provide such a list for flask?
That wasn't what I was suggesting. I'm saying that each repo owner should be maintaining such a list for their own repo, custom-tailored to it. Just like every repo owner manages their repo's .gitignore, .gitattributes, .dockerignore, etc.
And specifically, that list shouldn't be a set of things that are always filtered out of all searches, but rather a set of patterns that apply abstract purpose-type tags to files, ala .gitattributes, such that it makes them much easier (or even "default") to filter them in/out of your searches.
Given a patterns file that would look like:
*.test.js purpose=test
tests/** purpose=test
apps/*/tests/** purpose=test
(You could go wild from there, if you like, and let this patterns-file match not just files but lines inside files. For example, doc-comments lines [=~ lines with syntax X in files of type Y] could be tagged as having purpose=docs.)
And I would then propose, on top of that, some higher-level behavior — not necessarily implemented in rg(1) itself — that could look at a file of "filter-context specifications" like this:
default: code
code: +purpose=impl -purpose=test -purpose=examples -purpose=docs
The big win with those purpose-tags and scopes, would be if they had conventional or formal (e.g. URNs in an RDF namespace) names, such that different repos could agree on their usage.
Then tools that do code-search across many different repos, like Github, could layer UI on top of these purpose-tags and search-scopes discovered from the repo, to enable the repo to be searched using this explicitly-encoded abstract understanding of the purpose/function of the different parts of the repo. (Picture the Github search autocomplete for a search 'foo' giving several drop-down options like "Search 'foo' in [code] of this repo", "Search 'foo' in [docs] of this repo", etc. Then, after you get to the search page, a set of checkboxes to refine your search from the original scope, into a custom scope, by including/excluding arbitrary purpose-tags.)
https://ripgrep.datasette.io/-/ripgrep?pattern=with.*AsyncCl...