Elastic App Search would be well suited for something like that. It comes with a built in crawler.