Hacker News Reader

Comment by mritchie712 | Hacker News Reader

if you need to do really complicated file search, duckdb has an extension that lets you run SQL on your file system could be an option.

https://duckdb.org/community_extensions/extensions/hostfs.ht...

View on HN

Thanks! Search wasn’t the most difficult problem here. I found SQLite’s FTS5 to be quite powerful. Later combined with Tantivy, I think it’s a superb combination to tackle pretty much any local search needs.

What were the most difficult problems you faced, if it wasn’t search?

Getting everything to work reliably. This was my first time handling a software with so many moving parts. If I could get the search query to work, I would break the date range filter, if that worked, the document indexing flow would stutter, or I’d discover a new bug caused by an edge-case.

Since I built this application in an exploratory manner (branching out without a larger architectural design), it often became unwieldy for a solo hobbyist like myself.

Plus, I wasn’t able to really get going with user feedback. Reached out to family, friends and colleagues but faced the Windows roadblock and got demotivated. In the end, built it for myself and used it by myself. I am happy with that!

What issues did you have with FTS5 that made you switch to Tantivy? Does using Tantivy mean you have to load the entire Sqlite db into memory?

I ask because in my own project I have my own search query grammar [0] written in Lezer that more or less "transpiles" [1] to SQL+FTS5 that's been uh, tedious to build.

    [0] https://github.com/AlexErrant/Pentive/blob/main/shared-dom/src/language/query.grammar
    [1] https://github.com/AlexErrant/Pentive/blob/main/shared-dom/src/language/query2sql.ts

Duckdb can't even hanlde edge cases in a csv (pandas does it no problem) so I wouldn't trust it to do anything useful at the filesystem level

Reply on news.ycombinator.com