Hacker News Reader

Comment by sbochins | Hacker News Reader

What does the author mean by while indexing? You have to send a commit message to make the newly indexed doc searchable. Its silly to do this after every document. I'm assuming that is what the author was doing to get search poor search results. You could also just use Lucene's NRT abilities in Solr w/ a simple patch. I believe it is going to be working out of the box in the next version of solr. I wrote one myself to deal with real time search. If we want a fair comparison, we should be comparing Solr using NRT and ElasticSearch.

View on HN

It is not silly if you have asynchronous indexing because your content is generated by your users, one document at a time. If you want real-time indexing, that's the way to do it.

This is the default behavior for IndexTank. When an indexing call returns, whatever you indexed WILL show up in search results. We don't give you the option to commit or not commit, we simply designed IndexTank to be fast enough when indexing in real time. I'd like to see us included in that comparison as well.

What's IndexTank using as a backend?

IndexTank :) Meaning, we wrote a search engine from the ground up. We probably should write a blog post to explain why. We did use a couple of things from Lucene:

- The tokenizers, to be compatible with the query syntax - The index format for persistence on disk. It's a great format, no point in reinventing that. Analogous to how Open Office can save in .doc format.

Its not the only way to do it. Twitter doesn't do it that way, neither does LinkedIn. They have their own Search Engines as well build around Lucene. I haven't looked at them in a while, but I don't think they were commiting every document. They also get their content from their users.

Correct, Twitter in fact did a lot of work on the Lucene core specifically for this. The OP was talking about standard Solr though.

FYI, LinkedIn open-sourced the work they did to build real-time indexing/search on top of Lucene. It's called Zoie (http://javasoze.github.com/zoie/), I've played with it and it's nice from a performance point of view...

Lucene doesn't have to commit document updates to make them searchable. You just need to grab the IndexReader via the IndexWriter.getReader() method instead of opening an IndexReader on the index directory.

We use this for NRT indexing currently on our site and it works fine - there's a small (500 millisecond or less) delay in changes appearing but it's close enough for us.

We don't use Solr, though - it's a custom search engine based on Lucene.

NRT is just part of the story, since in order to provide a proper (near) real time features in a distributed architecture you need to build it around it. For example, not doing pull replication, but push replication. The whole distributed nature of elasticsearch compared to Solr was not discussed in this blog post, but its a big differentiator.

Reply on news.ycombinator.com