Have you investigated http://www.elasticsearch.org/ ?

I'd really recommend reading up on the Solr (I notice you mention Solr in your first post) and ElasticSearch projects; these guys, along with Lucene, have collectively solved many of the problems you're investigating.

They're both open source, and (Solr at least) have extensive mailing lists so you can see the sorts of problems people face when building generalized search engines.