Beating Google With CouchDB, Celery and Whoosh (Part 1)

http://andrewwilkinson.wordpress.com/2011/09/29/beating-goog... - part two actually contains the interesting content.

Nice post.

For those considering something like this, you might want to consider using scrapy, a Python web crawler, instead of rolling your own crawler.

I remember when I was looking for something like this a year ago and found that project. I've used it for a few things and it does a nice job abstracting away most of the core scraping architecture, but leaving room to be extended as necessary. After playing with it for a few weeks, makes you realize just how easy it is to grab a bunch of data from the web if you need it.

Whoosh is fast enough for small amounts of data, or when the searches do not need to be performed in realtime. But, I agree, something like Lucene/Solr or Sphinx might be a bit better for indexing large amounts of content (i.e., the provided example of a webcrawler indexing web pages).

Unfortunately, whoosh is too slow for any nontrivial amount of content. You're much better off using solr or ElasticSearch.