LinkedIn open sources IndexTank: search engine and service

Hey Emmet, we're one of the companies interested in continuing IndexTank's platform.

If you want to hear more, just ping me at emil@helpjuice.com

I looked at it some before IndexTank was bought (and I've done a reasonable amount of Solr work).

The biggest conceptual difference seemed to be that IndexTank was specifically written to autoscale - it was designed from the ground up to run on cloud providers, and to instantiate new resources as needed. It also has no central point of failure.

Solr Cloud (and things like Solandra) deliver some of this functionality to Solr.

I'd imagine they mean indexing (and being able to search on) data in real time. Given LinkedIn's previous open source projects around real time search (http://javasoze.github.com/zoie/).

Lucene (which Solr uses as its index) cannot expose newly indexed data immediately after it's added.

Lucene exposes IndexReaders for searches, which offer a snapshot view of the index. In order to search across new documents IndexReaders need to be re-opened, a somewhat expensive operation. Expensive enough to prevent it from happening after each document is added, especially if they're added frequently.

The latest version of Lucene supports "near real time" search, but afaik it's not widely used (with Solr).

Same here. Indextank service and pricing was great. Hoping someone can match it.

IndexTank provides real-time document indexing and its algorithm incorporates real-time metrics, like vote data. And it scales horizontally.

why did you find writing s custom scorer a pain? I've done it for raw lucene and it's trivial (in my case I added real time data in the formula using an external value source), I am not sure why it would be harder for Solr (I always got away with sorting order until now :).

IndexTank right now supports preffix search, stemming and a basic implementation of a Did You Mean feature. Regarding languages, it supports tokenization for every western language, and not long ago, we added support for CJK too.

Yeah, really though.

The indextank repo proper is interesting (and useful) enough, but indextank-service (https://github.com/linkedin/indextank-service) made my jaw drop a little. It's a full administrative stack for deploying indextank as a service.

It integrates anything that can be represented as a number. Prices, number of badges, importance of titles, it doesn't matter. You can combine any of those inputs into a relevance formula that is evaluated at query time. Of course IndexTank won't find those inputs for you, you have to provide them.

IndexTank categories are not designed for the tags use case, and will not work properly. It's intended for a relatively small amount of categories for which each document has a single value. The amount of different values of a category can be large but the amount of categories cannot. If you want to implement something like tags, then each tag should be a category because you'll want more than a single tag per document. We were in the process of designing a new feature to support this kind of use cases, and maybe we'll start a branch to implement it and hopefully the community will colaborate.