I ran into some significant performance issues with Bleve during a weekend hackathon a few months ago. I was trying to index the stackoverflow data dump for fun and I couldn't get it to successfully complete. I'm guessing i was running into some boltdb related limitations but I didn't have the time to dig deeper before the party ended and I had to get back to the day job.

SQLite's FTS5 allowed me to load the entire data set without breaking a sweat, but query performance was unexpectedly poor for some combinations of terms with no apparent consistency, and it became unusable due to a temp table I couldn't work out how to avoid when attempting to search in descending primary key order.

It was many months ago now and I have forgotten some of the particulars but thought it might make an interesting jumping-off point for discussion about any ideas anyone might have for indexing 60gb of data into a flat file setup like Bleve or SQLite uses.

Would the new scorch experiment perform better with an index of that size than the boltdb backend?